acceptodds
Under review as a conference paper at ICLR 2027

Flow-Matching-Based Protein Structure Tokenizer Made Efficient and Easy

Abstract

As the bridge between protein modality and discrete modeling, protein structure tokenization still largely relies on heavily engineered training objectives tailored to specific downstream tasks and large training datasets, which hinders its transfer to broader application scenarios. To address this issue, we propose ProFiT, a lightweight flow matching tokenizer. With simple training strategies that encourage healthy codebook utilization, ProFiT can be trained efficiently and naturally learns semantically meaningful representations without any manual semantic alignment, while achieving reconstruction quality and generalization that match or surpass those of substantially larger tokenizers. We conduct extensive evaluations across a wide range of settings and demonstrate that ProFiT is a plug-and-play tokenizer adaptable to diverse downstream tasks. This study further reveals the significant potential of the flow matching tokenizer paradigm. Our code is publicly available at https://anonymous.4open.science/r/ProFiT-5DD9.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein structure tokenizer, flow matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Hss7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new diffusion tokenizer for protein structures. The paper benchmarks these methods on reconstruction and generative models, and demonstrates strong reconstruction and generative performance. The reconstruction benchmarks are on standard held-out test sets; they train both diffusion and autoregressive models to validate their approach. They also benchmark their tokens on StructTokenBench.

Critical strengths

The results are broadly strong and compelling. Both ODE and annealed SDE performance seem good on reconstruction. The generative tasks are standard and sensible. On a first shallow pass, the paper does a good job convincing me that their approach to tokenization has value. I've expanded on qualifications to this in the critical weaknesses section.

Critical weaknesses

There are two main issues I have with this paper.

  1. Comparisons to Kanzi/DiffAE: This paper is about diffusion autoencoders. Kanzi and ProteinAE were (to my knowledge) the first biological diffusion autoencoders, and for that reason I think the comparison to them is the most important. The paper makes multiple changes in methodology compared to these works (e.g., wrt Kanzi, FSQ -> VQ-VAE, no sliding window attention, a different encoder/decoder setup, etc). My view is for the methodological contribution to be useful, the paper needs to actually isolate what's important and what isn't. This leads to the following issues:

a) Really, every comparison that includes Kanzi should include ProteinAE. Given the relatively short period for rebuttal, I would be willing to move to acceptance without this if the Kanzi comparisons were made much more rigorous, but in my opinion there is not a ton of value in comparing to the main tokenizers besides DPLM2 and ESM3 (most of the others aren't really in use).

b) The Kanzi results in reconstruction look extremely poor; in the appendix, this is attributed to length generalization, where CAMEO contains longer proteins than Kanzi is trained on. I'd prefer to see length-filtered CAMEO results in the main text (alongside the current results), so I can compare how much of this performance difference is due to length generalization (i.e., if I could just train a better kanzi with longer proteins then the results are less interesting).

c) Similarly, the generative results for Kanzi need to be more carefully done. Are the poor generation results because of some issue with Kanzi, or because you're training an AR model to output tokens longer than the tokenizer decoder is compatible with.

d) The architectural and training/data differences need to be more carefully compared against Kanzi + ProteinAE. Linking back to (a), it's extremely interesting if the method does generalize better to longer proteins despite being trained on the same data, but we need to have an idea of which changes are actually inducing this improvement. There's too much going on right now (architecture + timestep sampling etc) for me to tell. This is a critical issue for the paper; simply providing a better tokenizer without arguing what is driving the improvements is insufficient for acceptance in my view.

e) This is somewhat opinionated, but I don't think the StructTokenBench results are too important. As a benchmark, I don't think it correlates well with actual tokenizer performance. So if no other reviewers have issues here, I'd encourage the authors not to spend too much time there.

d) I'm somewhat skeptical of some of the discrete diffusion results. DPLM seems way too high -- having used that model, 98% seems really high for designability.

  1. What's the value of tokenization?: The ML for proteins field has broadly moved away from tokenization of protein backbones. We've moved beyond tasks like backbone structure generation towards all-atom generation and sequence-structure codesign, which makes it hard to get excited about a new way of doing backbone tokenization. Tokenization still has its uses (e.g., for representing sidechains as in la proteina), but we don't need compression like we do for vision and in general we can just represent the full protein sequence, since proteins aren't that long compared to modern context windows. This is all to say, I think protein structure tokenizers need to actually demonstrate an advantage of tokenization, as opposed to appealing to ways in which tokenization could be useful (e.g., combining with natural language).

Summary: I would support acceptance if the advantages of this relative to Kanzi were clearly enumerated, with maybe some discussion of the differences to ProteinAE in the Appendix. (2) is important but not a nonstarter for acceptance.

Questions for authors
  1. What change -- architectural, training, data, etc -- is actually driving the performance differences relative to Kanzi?
  2. What's the relative performance once length generalization is removed?
10 comments
AuthorsOpenReview ↗

Response to Reviewer Hss7 (1/n)

W1 & Q1 & Q2: Additional Baselines, Length Generalization, and Evaluation Details

We thank the reviewer for these detailed suggestions. We address each point below.

a) Comparison with ProteinAE

Kanzi and ProteinAE are closely related works. Kanzi encodes protein Cα structures into discrete tokens using FSQ and reconstructs structures with a flow matching decoder. ProteinAE learns continuous hidden representations of protein backbones and reconstructs structures through flow matching decoding. Since only Kanzi is a discrete tokenizer and thus belongs to the same category as ProFiT, our paper compares ProFiT with Kanzi and other tokenizers, without including ProteinAE. Nevertheless, we agree that comparisons with continuous autoencoders can help readers understand how much structural fidelity is lost during discretization and how much room remains for improving tokenizers. We therefore provide additional reconstruction results for ProteinAE below, reported as RMSD / TM-score:

MethodCAMEO2022CASP15CASP16D-FS
ProteinAE (20 steps)0.183 / 0.9980.374 / 0.9940.268 / 0.9970.104 / 0.999
ProteinAE (100 steps)0.188 / 0.9980.397 / 0.9930.276 / 0.9970.119 / 0.999

Because ProteinAE reconstructs structures from continuous hidden representations without a discrete bottleneck, its reconstruction performance substantially exceeds that of all tokenizer baselines. Its performance is also consistent with our observations in Figure 4 regarding the degree of distribution shift across test sets. Both ProFiT and Kanzi lose considerable reconstruction accuracy at the discrete bottleneck. This highlights a central challenge for structure tokenizers: preserving the geometric information in continuous representations with a codebook of limited capacity.

We thank the reviewer for suggesting this additional baseline and will include these results in the revised manuscript.

b) Reconstruction under the Same Length Truncation

Appendix C.1 provides an additional discussion of Kanzi's length generalization and reports its reconstruction errors on CAMEO2022, CASP15, and CASP16 after length truncation. Here, we additionally report ProFiT's reconstruction performance under the same truncation protocol. The metrics are Cα-RMSD / TM-score:

MethodCAMEO2022CASP15CASP16D-FS
ProFiT (ODE, 20 steps)0.674 / 0.9790.727 / 0.9831.002 / 0.9740.503 / 0.983
Kanzi (default, 100 steps)0.993 / 0.9400.932 / 0.9591.341 / 0.9390.992 / 0.943
Kanzi (20 steps)1.477 / 0.9051.863 / 0.9042.346 / 0.8821.426 / 0.891

Although Kanzi's performance improves substantially with length truncation, ProFiT still achieves substantially better reconstruction performance across all test sets under the same evaluation protocol. We attribute this advantage to differences in model architecture and the training stabilization techniques described in Section 4.2. For a more detailed analysis of the architectural factors underlying these results, please refer to our response to point d).

AuthorsOpenReview ↗

Response to Reviewer Hss7 (2/n)

c) Sequence Lengths in the Generation Experiments

For the AR experiments in the unconditional generation evaluation in Section 5.2, Appendix B.3 specifies that the AR model uses the same training set as ProFiT: AFDB Foldseek-cluster data with sequence lengths ≤256\leq 256 residues and pLDDT scores ≥75\geq 75. We also set max_token = 256 during inference. Thus, the AR evaluation in Table 2 stays within Kanzi's suitable length range.

For DPLM generation, as described in Section 5.2, we sample 50 sequences at each length of 100, 200, 300, 400, and 500 residues. Some evaluated sequences therefore exceed the length range in which Kanzi performs well. We consider this a fair comparison because generalization across sequence lengths is an important property of a tokenizer, and Kanzi's architecture does not preclude evaluation on longer sequences. Nevertheless, we provide Kanzi's results for lengths of 100 and 200 residues below and will include them in the revised manuscript. These results help clarify whether Kanzi's performance gap primarily arises from the evaluated sequence lengths or from its suitability for the generation setting itself.

For the conditional generation experiments in Section 5.3, we use CAMEO2022 as the evaluation set without filtering by length. For the same reasons, we will retain the current results in the main text and additionally report results restricted to proteins with ≤256\leq 256 residues.

Unconditional generation:

Length settingscRMSD ↓\downarrowscTM ↑\uparrowDes. ↑\uparrowDiv. ↓\downarrow
Kanzi (L=100L = 100)7.29±0.787.29 \pm 0.780.53±0.030.53 \pm 0.030.65±0.060.65 \pm 0.060.40±0.010.40 \pm 0.01
Kanzi (L=200L = 200)18.20±0.4418.20 \pm 0.440.34±0.010.34 \pm 0.010.12±0.040.12 \pm 0.040.27±0.020.27 \pm 0.02
Kanzi (Avg.)12.75±0.3312.75 \pm 0.330.44±0.010.44 \pm 0.010.38±0.050.38 \pm 0.050.33±0.010.33 \pm 0.01

Conditional generation:

ModelRMSD ↓\downarrowTM-score ↑\uparrow
Kanzi (L≤256L \leq 256)11.66±0.0211.66 \pm 0.020.38±0.000.38 \pm 0.00

In unconditional generation, restricting the evaluation to L=100L = 100 and L=200L = 200 does not substantially change Kanzi's average metrics. Moreover, generation quality already degrades markedly at L=200L = 200, supporting the view that Kanzi's tokens are less well suited to DPLM generation. In conditional generation, length filtering slightly improves Kanzi's performance but does not materially change its ranking on these benchmarks. These two additional experiments suggest that more fundamental issues in Kanzi's tokenizer learning process limit its performance on generative tasks.

We thank the reviewer for prompting this clarification of our evaluation settings. All additional information provided here will be included in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Hss7 (3/n)

d) Investigating the Source of Length Generalization

We thank the reviewer for this question. We agree that the current manuscript does not sufficiently explain why ProFiT generalizes well across sequence lengths. Appendix.D already provides ablations of design choices, sampling steps, and codebook size, but these factors do not explain ProFiT's length generalization. Here, we provide an additional explanation: the positional treatment of token conditioning in ProFiT's decoder. This implementation is inherited from ProteinAE.

In Kanzi's decoder, noisy coordinate embeddings and token embeddings are concatenated along the sequence dimension. For a protein of length LL, this produces a sequence of length 2L2L. Self-attention then generates QQ, KK, and VV from this sequence, with RoPE applied to QQ and KK. Consequently, the noisy coordinates at position ii correspond to token conditioning at position L+iL + i.

In ProFiT and ProteinAE, tokens instead serve as conditioning inputs. The latent representation ziz_i associated with a structure token is first projected into the conditioning channels and incorporated into the conditioning features for residue ii. These features then modulate, through AdaLN, the representation sis_i constructed from that residue's noisy backbone coordinates xt,ix_{t,i}. RoPE is subsequently applied to QiQ_i and KiK_i generated from these condition-modulated representations. Thus, the conditioning is aligned with the noisy coordinates of the same residue, with this correspondence established through residue-wise conditioning injection.

We reiterate that this decoder design originates from ProteinAE and is not our original contribution. Similar designs appear in work such as DPLM2, where positional encodings are aligned across tokens from different modalities rather than assigned through simple concatenation. We therefore hypothesize that this design may be key to length generalization.

To test whether this change improves Kanzi, we initialized from Kanzi's pretrained checkpoint, modified only how RoPE is applied, and continued training on the same dataset. We observed a substantial improvement on proteins with >256> 256 residues, without a substantial change in reconstruction performance on proteins with ≤256\leq 256 residues:

Model≤256\leq 256 residues>256> 256 residues
Kanzi (original, 20 steps)0.993 / 0.94020.63 / 0.320
Kanzi (modified, 20 steps)1.078 / 0.9493.035 / 0.791

However, training converged at approximately this performance level, and further training did not yield substantial improvements. We also conducted an additional experiment that replaced ProFiT's decoder design with one resembling Kanzi's and observed a very similar sharp degradation on long sequences. We therefore attribute the remaining gap between modified Kanzi and ProFiT to the other architectural choices and training stabilization techniques proposed in this work.

We will add this explanation to the revised manuscript and clearly distinguish the contributions originating from different sources. We thank the reviewer for encouraging this additional analysis.

e) The Role of StructTokenBench

We thank the reviewer for this suggestion. We initially included this benchmark as a proxy for a range of downstream tasks. Given the breadth of downstream tasks and the limited maturity of generative models operating on protein structure tokens, stable training results cannot yet be guaranteed across all such tasks. StructTokenBench provides a quick and straightforward way to assess the semantic information retained in tokens. We have kept Section 5.4 as concise as possible and have decided to retain it in its current form for completeness. We agree that downstream utility should ultimately be assessed on the specific downstream tasks themselves.

Reviewer_Fd5kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProFiT, a protein structure tokenizer combining vector quantization with a conditional flow-matching decoder. The framework uses token masking and dead-code replacement to improve training robustness and codebook utilization. Experiments demonstrate strong reconstruction accuracy, out-of-distribution generalization, and competitive performance across autoregressive and discrete diffusion generation, conditional folding, and selected functional-site prediction tasks.

Critical strengths

Strong reconstruction performance. ProFiT achieves impressive reconstruction accuracy across multiple benchmarks, including challenging out-of-distribution structures, while maintaining high codebook utilization.

Practical and relatively simple training framework. The combination of flow matching with straightforward codebook stabilization techniques provides a practical alternative to more heavily engineered tokenizer objectives. The evaluation across autoregressive and discrete diffusion generation is also valuable.

Critical weaknesses
  1. Incomplete positioning against relevant tokenization approaches and incomplete downstream evaluation. The paper omits Protein Structure Tokenization via Geometric Byte Pair Encoding (GeoBPE, ICLR 2026), despite presenting a broad characterization of the protein structure tokenization landscape. GeoBPE offers an alternative paradigm based on hierarchical geometric tokenization rather than fixed residue-level neural quantization, with demonstrated data efficiency, compression, and downstream transfer. Its omission from Section 2.2 limits the completeness of the related-work discussion and the context for ProFiT's generality claims. The paper evaluates only four functional-site prediction tasks from StructTokenBench (Yuan et al., 2025), despite the availability of a broader evaluation suite. Relevant baselines, including GeoBPE, inverse-folding representations, and AminoAseed, are also absent. Since transferability and general-purpose applicability are central claims, the restricted task coverage and baseline selection make it difficult to establish whether ProFiT consistently outperforms alternative tokenization approaches beyond reconstruction and generation.

  2. Insufficient motivation for flow matching as a general-purpose tokenization paradigm. Although the reconstruction improvements are convincing, the broader argument for adopting flow matching remains underdeveloped. High codebook utilization and accurate reconstruction are useful intermediate properties, but do not necessarily translate into better downstream representations. The evidence for plug-and-play transferability is mixed: ProFiT performs competitively across generative architectures, but trails specialized baselines on several generation metrics and does not consistently outperform existing tokenizers on functional-site prediction (Table 4). Thus, the experiments support ProFiT as a strong reconstruction-oriented tokenizer more convincingly than as a generally superior representation for diverse downstream tasks.

Questions for authors
  1. Why does the StructTokenBench evaluation cover only four functional-site tasks? Could the authors extend Table 4 to the broader benchmark and include relevant baselines such as GeoBPE, AminoAseed, and inverse-folding representations?

  2. What evidence establishes that flow-matching-based tokenization offers advantages beyond reconstruction accuracy and codebook utilization? In particular, how do the authors reconcile the claimed plug-and-play generality with the mixed downstream results in Tables 2–4?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Fd5k (1/2)

W1 & Q1: Additional Baselines and Expanded StructTokenBench Evaluation

We thank the reviewer for these suggestions. GeoBPE and AminoAseed were indeed missing from Section 2, and we will include them in the revised manuscript. During encoding, GeoBPE merges adjacent fragments into motifs of varying sizes based on local geometry and learned rules. Consequently, protein structures of the same length can yield token sequences of different lengths. This poses substantial challenges for training both the AR model and DPLM2, making GeoBPE difficult to include in our comparisons on generation tasks.

We first provide additional reconstruction results for AminoAseed and GeoBPE:

CAMEO2022 RMSDCAMEO2022 TMCASP15 RMSDCASP15 TMCASP16 RMSDCASP16 TMD-FS RMSDD-FS TM
AminoAseed5.830.68315.390.52213.780.4588.030.510
GeoBPE5.870.5808.010.5587.810.5744.230.578

For AminoAseed, we found that the authors' RMSD implementation includes an additional division by 3\sqrt{3} relative to ours and permits reflections during structural alignment. Our evaluation protocol, like those used in most structure tokenizer studies, does not permit reflections. We therefore evaluate AminoAseed using the same protocol as our benchmark. As a result, the values above differ from those reported by AminoAseed but allow a fair comparison with Table 1. Neither baseline shows a clear improvement in reconstruction accuracy over the existing baselines.

Regarding the StructTokenBench evaluation in Table 4, these experiments primarily examine whether ProFiT's structure tokens retain information useful for downstream prediction without an auxiliary loss for explicit semantic alignment. We largely agree with Reviewer Hss7's observation in Weakness 1(e) that StructTokenBench's evaluation settings differ somewhat from practical downstream scenarios, such as folding with DPLM2. Consequently, tokenizer performance on this benchmark may not strongly correlate with performance in those downstream applications. This motivated the limited length of Section 5.4 and the smaller scope of the benchmark evaluation presented there.

Nevertheless, we agree that a broader evaluation would improve the completeness of our work. We therefore expand the StructTokenBench evaluation to cover as many settings as possible and consider GeoBPE and AminoAseed as additional baselines. We also include the encoder of the inverse folding model ProteinMPNN as an inverse folding representation baseline, to assess the performance achievable on StructTokenBench without a discrete bottleneck. We will include these additional experiments in the revised manuscript. Because GeoBPE does not produce token sequences with one token per residue, a fair comparison on residue-level classification tasks is difficult; we therefore exclude it from the StructTokenBench evaluation.

ProteinMPNNFoldSeekProTokensESM3AminoAseedProFiT
BindIntFold48.9853.1844.6644.3047.1144.97
SupFam93.1646.2086.0590.7790.5381.44
BindBioFold77.9852.3758.4762.8465.7357.49
SupFam80.0452.4160.4765.2268.3058.38
BindShaketest75.9253.4059.8266.1069.6165.61
CatIntFold63.0053.4358.1661.0962.1954.91
SupFam90.2151.4183.8589.8291.9175.33
CatBioFold84.8256.3756.1465.3365.9559.99
SupFam90.4253.7864.0574.6587.5966.29
ConFold59.1149.2656.2355.2257.2356.94
SupFam82.6351.3974.3380.5386.6079.19
RepFold77.3947.7077.2574.7074.9775.30
SupFam81.6052.5378.9082.3684.5779.42
EptFold65.9354.5254.6963.6962.1655.11
SupFam75.9350.5667.5261.9772.0263.31
FlexRMSFFold43.7415.3513.8144.5344.6316.35
SupFam43.9511.997.6239.6840.9915.48
FlexBFactorFold27.754.176.6723.6021.3012.99
SupFam26.296.975.4725.8021.7612.68
FlexNEQFold46.195.7112.9845.0849.643.24
SupFam44.492.6012.5045.4350.153.87
HomoFamily76.9715.3018.3354.4269.7817.56
Fold21.9611.575.8430.0229.8714.00
SupFam46.174.676.1724.8938.3813.97

Overall, ProFiT does not clearly outperform ESM3 or AminoAseed on this benchmark, but consistently outperforms FoldSeek and performs comparably to ProTokens. Across task categories, ProFiT performs relatively well on Functional Site Prediction, while its performance in the other categories is substantially lower than that of ESM3 and AminoAseed. We reiterate that no semantic auxiliary loss is used during training: neither its performance on these semantic tasks nor its relative strength in Functional Site Prediction was explicitly optimized for. We will include the complete benchmark results in the appendix of the revised manuscript. We thank the reviewer for this suggestion.

AuthorsOpenReview ↗

Response to Reviewer Fd5k (2/2)

W2 & Q2: Advantages of Flow Matching-Based Tokenizers

We thank the reviewer for this suggestion. We believe that the value of flow matching-based tokenization lies in two complementary aspects: reusing the same structural representation across downstream architectures and adjusting the computational budget for geometric decoding as needed. High reconstruction accuracy and codebook utilization serve as diagnostic metrics for this design; its downstream utility must still be validated through actual tasks. Tables 2–4 further examine its generative capabilities, cross-modal applicability, and retention of functional information.

First, Table 2 supports ProFiT's applicability across architectures. ProFiT achieves 83% designability in unconditional generation under both AR and DPLM2 architectures, compared with 71% and 98% for the DPLM2 tokenizer and 84% and 74% for ProTokens, respectively. The conditional generation experiments yield similar observations. These results indicate that specialized representations have their respective strengths, while ProFiT maintains consistent and competitive performance across modeling paradigms. This underpins our claim of plug-and-play applicability: the same tokenizer and discrete interface can be reused, with different downstream models learning the corresponding token distributions. Some recent applications, such as speculative decoding, also call for data representations compatible with multiple modeling paradigms, and ProFiT provides a useful tool for exploring such applications.

Furthermore, the flow matching decoder offers an adjustable inference compute budget, which is another key motivation for adopting this paradigm. The tokenizer's geometric encoding and decoding are designed to be largely independent of the downstream prior architecture, such as AR or DPLM2. With the tokenizer weights, token interface, and downstream prior held fixed, users can adjust the number of decoding steps to allocate computation in continuous geometric space according to task requirements. Appendix D.2 provides empirical evidence: a small number of sampling steps already achieves high reconstruction accuracy, while increasing the number of steps from 5 to 20 on the more challenging CASP15 benchmark improves RMSD from 2.73 Å to 1.48 Å. Table 15 further reports ProFiT's computational cost, showing that it reconstructs 3D coordinates with substantially less computation than Kanzi under both single-step and multi-step decoding. Users can therefore choose an appropriate budget for rapid conformation checks or more precise geometric reconstruction.

This property also provides a natural interface for future reasoning with structure tokens. For example, in a multi-step reasoning process in token space analogous to LLM chain-of-thought, token representations could be retained when coordinates are unnecessary, decoded with a few steps when intermediate conformations need to be inspected, and decoded with more computation for the final structure output or accuracy-sensitive structure prediction. This potential application lies beyond what we can validate in this paper, but we consider it a promising direction for future exploration.

We agree with the reviewer that ProFiT's advantage is more pronounced in reconstruction than in downstream tasks. We will strengthen the motivation accordingly: ProFiT contributes not only high-fidelity structure compression, but also reusable representations across generative architectures and a geometric decoding mechanism whose computational budget can be flexibly adjusted independently of the downstream prior.

Reviewer_Fna3OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

ProFiT is a protein-backbone tokenizer with a vector-quantized bottleneck and a conditional flow-matching decoder. Its training recipe uses random token masking and dead-code replacement to improve robustness and codebook utilization. The paper evaluates reconstruction on four datasets, unconditional generation using both autoregressive and discrete-diffusion priors, conditional folding, and functional-site prediction. It also studies component ablations, sampling steps, vocabulary size and computational cost, and analyzes the length sensitivity of the Kanzi baseline.

Critical strengths
  1. Strong reconstruction is supported by several complementary evaluations. Table 1 reports SDE reconstruction RMSDs of 0.64, 1.19, 1.01 and 0.42 Å on CAMEO2022, CASP15, CASP16 and the held-out D_FS subset, respectively, with high TM-scores. The analysis goes beyond an aggregate comparison: Appendix C.1 separates Kanzi's results within and beyond 256 residues, and Appendix D.3 repeats evaluations at K=1000 to address vocabulary-size differences. These checks strengthen the evidence for a useful reconstruction model, including outside the training-length regime.
  2. The training recipe and evaluation breadth make the work practically useful. A single tokenizer is evaluated with both AR and discrete-diffusion generation, conditional folding and functional-site prediction. It is not the best tokenizer on every task, but the combined profile is attractive. Table 9 supports both proposed training components: removing masking or code replacement worsens reconstruction and AR self-consistency. Appendix B.2 reports a concrete training budget of approximately 20 hours on eight H20 GPUs, and Appendix D.4 provides a computational comparison with Kanzi. The contribution is a useful recipe and empirical study rather than the first use of flow matching for protein tokenization.
Critical weaknesses
  1. Some experimental results are not sufficiently traceable to a single evaluation configuration. Table 9 reports a full-model CAMEO reconstruction RMSD of 0.84, whereas Table 1 reports 0.96 for ODE and 0.64 for SDE. Table 14 reports K=8192 folding results of 8.31 Å / 0.787 TM-score, whereas Table 3 reports 8.61 Å / 0.724. Different seeds, sampling settings or aggregation could explain these differences, but that mapping is not supplied. Moreover, the five-run generation results do not clearly establish whether the tokenizer and downstream priors are independently retrained or only resampled; Section 5.3 specifically describes repeated evaluation. This matters for judging reproducibility, ablation consistency and the uncertainty of cross-tokenizer comparisons.
  2. Some interpretations of generality and semantic quality go beyond what the experiments directly isolate. Table 2 favors ProTokens on AR self-consistency and DPLM2 on diffusion generation, and Table 3 favors DPLM2 on folding. Table 4 also shows mixed functional-site performance, including below-50 AUROC for ProFiT on the BindInt Fold split. These results support useful but task-dependent representations rather than uniformly strong semantic understanding. In addition, Figure 5 varies sampling temperature and plots scRMSD; it does not directly measure whether likelihood ranks individual generated structures by quality, as suggested in the accompanying text. These qualifications do not negate the reconstruction contribution, but are important for the broader claims about representation quality and screening utility.
Questions for authors
  1. Can you map Tables 1, 3, 9 and 14 to their exact checkpoints, sampler settings, random seeds, test instances and aggregation rules, and explain the numerical differences identified above?
  2. Which reported standard deviations reflect independent tokenizer/prior training, and which reflect repeated decoding from fixed checkpoints? Please specify the shared downstream training budget/configuration and the validation criterion used to choose sampler settings across tokenizers.
  3. Is there a per-sample, length-controlled likelihood-versus-quality analysis supporting the screening claim? If not, would you narrow that claim to the temperature-dependent quality trend actually shown in Figure 5 and describe the functional-site evidence as task-dependent? This affects the interpretation of the claimed semantic benefit.
3 comments
AuthorsOpenReview ↗

Response to Reviewer Fna3 (1/2)

W1 & Q1: Missing Evaluation Configuration Details

We thank the reviewer for pointing out the inconsistencies in the reported numbers and the missing evaluation configuration details.

For ProFiT's reconstruction metrics, different noise samples can lead to slight variations across reconstruction runs. Table 1 therefore reports the mean results over five reconstruction runs with different random seeds using the same checkpoint, with standard deviations provided as subscripts. In contrast, Table 9 in the submitted manuscript reports the results from the first of these runs. We apologize for this inconsistency. To ensure accuracy, we will update the corresponding values in Table 9 to the means over all five runs in the revised manuscript.

For ProFiT's conditional folding metrics, Table 3 reports the mean results over five independent DPLM sampling runs for each sequence, whereas Table 14 reports the results from the first of these runs. We will also update Table 14 to report the means over all five sampling runs in the revised manuscript.

As described in Section 5 and Appendix B.1, all experiments except the ablation studies use the checkpoint trained for 120k steps with the configuration specified in Appendix B.1. Table 9 uses the ODE sampler (γ=η=0\gamma = \eta = 0), while Tables 3 and 14 use the SDE sampler (γ=η=0.25\gamma = \eta = 0.25), consistent with Section 5.2. We will clarify and supplement these details in the revised manuscript. We will also check the other tables, including those not mentioned by the reviewer, to ensure consistency across all tables. We thank the reviewer for these helpful suggestions.

W1 & Q2: Clarification of Standard Deviations, Downstream Training Configurations

We thank the reviewer for requesting clarification on these details. The standard deviations reported across our experiments are computed over multiple decoding runs with different random seeds using a single fixed checkpoint. Regarding downstream training configurations, Section 5.2 and Appendix B.3 describe the training procedure and configuration for the AR model. We use the same AR training procedure for all baselines and ProFiT, with the training data constructed using the respective tokenizers. Sections 5.2 and 5.3 describe the specific configurations used to train DPLM2. We train DPLM2 using the official training scripts and default training configuration. As with the AR model, we construct the training data using the respective tokenizers.

Tokenizers that are not based on flow matching are generally deterministic and do not require sampler selection. For the Kanzi baseline, we use the weights and default sampler provided in its official repository. For ProFiT, Sections 5.1 and 5.2 report the sampler parameters (η\eta and γ\gamma) used in each experiment. These hyperparameters were selected through grid searches for the different downstream tasks.

We will further emphasize and supplement these details in the revised manuscript. We thank the reviewer for helping us improve the completeness of our presentation.

AuthorsOpenReview ↗

Response to Reviewer Fna3 (2/2)

W2: Clarifying the Scope of Generality and Semantic Information

We thank the reviewer for highlighting the need to clarify the scope of our conclusions. By “generality” we mean the applicability of the same structure discretization scheme across different modeling paradigms and use cases. Our experiments also show that different methods perform best on different tasks, which is consistent with this claim.

Table 2 provides direct evidence of performance across paradigms. For example, ProFiT achieves 83% designability under both AR modeling and discrete diffusion, compared with 84%/74% for ProTokens and 71%/98% for DPLM2, respectively. We therefore acknowledge the advantages of ProTokens and DPLM2 in their respective generation settings, while ProFiT's results support its relatively balanced applicability across paradigms. Regarding semantic quality, our conclusion is that the learned representations retain task-relevant information that supervised predictors can use, even though tokenizer training uses neither functional labels nor an auxiliary loss for explicit semantic alignment. We agree that ProFiT does not achieve the strongest performance on this benchmark. Please also see our response to Reviewer Fd5k's W1 & Q1, where we provide the complete StructTokenBench evaluation. We will refine the phrase “semantic understanding” to state that “ProFiT provides representations containing semantic information whose utility varies across downstream tasks.”

We will therefore clarify our conclusion as follows: ProFiT provides a unified structure tokenization interface applicable to multiple generative paradigms and downstream uses, while retaining task-relevant information in its tokens. This statement preserves the core contribution of our work and is consistent with the task-dependent performance shown in Tables 2–4. We agree that performance on specific downstream tasks must be established empirically, and further evaluation can be pursued in future work.

W2 & Q3: Likelihood-versus-quality analysis

We thank the reviewer for this experimental suggestion. Figure 5 shows the relationship between sampling temperature and mean scRMSD. We additionally evaluate the same sets of AR-generated structures using several fixed scoring temperatures and compute the length-normalized NLL to account for differences in generated sequence length:

= -\frac{1}{L} \sum_{i=1}^{L} \log p_{\theta} \left(z_i \mid \text{BOS}, z_1,\ldots,z_{i-1}\right).$$ For each sampling temperature, we compute the partial Spearman correlations between the AR model's NLL and the scRMSD and scTM metrics, controlling for sequence length and generation seed: | Sampling temperature | Partial $\rho(\mathrm{NLL}, \mathrm{scRMSD})$ $\uparrow$ | Partial $\rho(\mathrm{NLL}, \mathrm{scTM})$ $\downarrow$ | |---:|---:|---:| | 0.2 | −0.017 | 0.012 | | 0.3 | 0.400 | −0.519 | | 0.5 | **0.810** | **−0.809** | | 0.7 | 0.742 | −0.770 | | 1.0 | 0.587 | −0.634 | At sampling temperatures between 0.3 and 1.0, these results suggest that NLL can serve as a proxy for downstream structural quality. This supports using NLL to preselect promising samples in large-scale AR generation, potentially improving generation efficiency. A similar observation is discussed in Appendix.D of Kanzi [1], which describes this as “a free estimator for the downstream model quality.” At a sampling temperature of 0.2, however, the correlations between NLL and the structural metrics are nearly zero, in contrast to the results at temperatures between 0.3 and 1.0. At very low temperatures, the sampling distribution becomes more concentrated, potentially compressing the variation in both NLL and structural quality and weakening NLL's ability to distinguish sample quality. Empirically, sampling protein sequences or structures at very low temperatures is also generally undesirable: the generative model tends to repeat patterns from the training set, substantially reducing diversity and novelty. We will include this experiment in the revised manuscript to demonstrate the practical utility of NLL in AR generation and report our additional findings at low sampling temperatures. [1] Dilip, Rohit, et al. "Flow autoencoders are effective protein tokenizers." International Conference on Learning Representations, 2026.

Thank you for clarifying the experiment settings, fixing the inconsistent results, and providing additional analysis. They addressed some concerns. I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.