acceptodds
Under review as a conference paper at ICLR 2027

TACFormer: Transformer-Based Generative Model for Target-Conditioned De Novo PROTAC Design

Abstract

Targeted protein degradation (TPD) represents a fundamental shift in drug discovery, moving beyond traditional target inhibition toward the event-driven removal of disease-causing proteins. A prominent strategy for TPD is the use of proteolysis-targeting chimeras (PROTACs), heterobifunctional molecules that selectively route a protein of interest (POI) to the ubiquitin-proteasome system for degradation. While PROTAC design still relies heavily on trial-and-error synthesis and experimental testing, deep generative models offer a promising means to explore PROTAC chemical space in silico and generate novel molecules. However, many existing generative methods formulate PROTAC design primarily as a linker-generation problem, with POI-binding (warhead) and E3-ligase-binding components fixed in advance. This restricts exploration of the full PROTAC chemical space and often leaves the POI implicit through the predefined warhead, rather than using protein-level information as an explicit conditioning signal during generation. To address these limitations, we propose TACFormer: a transformer-based generative model for target-aware de novo PROTAC design. The model conditions a GPT-style molecular decoder on pretrained ESM-2 representations of POI sequences, allowing POI-level information to steer full-molecule generation. Our results show that TACFormer generates highly valid, novel, and chemically diverse PROTACs while showing consistent POI-dependent shifts in generated chemical space. In particular, molecules generated under specific target conditions preferentially populate chemical neighborhoods associated with POIs in the held-out dataset, while extrapolating to novel chemotypes. By natively integrating biological context into the generative process, TACFormer moves PROTAC design toward a more biology-informed exploration of degrader chemical space.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Machine Learning, Deep Generative Models, Transformers, Molecular Design, AI for Science, Drug Discovery, Conditional Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wPM9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TACFormer generates PROTAC-like molecules for a target protein by gating pooled ESM-2 features into a decoder pretrained on 79,406 molecules (Section 3.3, Eq. (1)). Across 20 seen targets, held-out likelihoods and neighbor purity show clear chemical steering, but transfer fails on unseen WDR5 and DHFR. Section 6.1 limits claims to chemical distributions rather than bioactivity, yet I lean toward a weak reject until the conditioning and split questions are resolved.

Critical strengths

Chemical steering on seen targets rests on several checks. Table 1 shows positive mean ΔNLL\Delta_{\mathrm{NLL}} on all 20 POIs with a 0.3580.358 macro mean, and macro purities of 82.11%82.11\% at k=5k=5 and 78.37%78.37\% at k=10k=10 (and 72.41%72.41\% at k=20k=20) while accounting for alternative target labels in Section 4.3 and Appendix B.8.

I also appreciate that the authors are candid about failure modes. Figure 3 checks target clusters inside separate CRBN and VHL subsets to rule out an E3 ligase confound, while Table 1 reports METTL14 mode collapse and weak SMARCA4 purity, and Table 2 shows negative ΔNLL\Delta_{\mathrm{NLL}} on unseen targets.

Critical weaknesses
  1. My main worry is that the paper never tests whether ESM-2 features beat a simple target ID. In Sections 3.2 and 3.3, each seen POI is a fixed projected vector, so the decoder can learn per-target distributions without sequence information. The unconditional baseline in Section 5.1 does not test this. Since Table 2 transfer fails and Chen and Bajorath (2024) already conditioned SMILES transformers on protein embeddings, I wasn't convinced ESM-2 does the work without a target ID or random vector control.
  2. I also couldn't tell if the splits rule out memorizing close analogs. Appendix B.3 uses a random 80/10/10 split of molecule and POI pairs without stating if shared canonical molecules stay in one split or if test molecules were removed from the 79,406 ZINC and ChEMBL pretraining set. Even disjoint molecule splits put close analogs across train and test sets under random splitting, which matters given the tight clusters in Figure 3 (and the anonymous repo link returned HTTP 403).
  3. Table 3 does not show new target-specific warhead chemotypes. String uniqueness in Eq. (2) and the 89.18%89.18\% unseen warhead rate only check against training warheads for that one target, so copying a training warhead from another target still counts as new while purity is only checked on whole PROTACs. Also, Section 5.1 mismatches pooled unconditional diversity (0.7900.790) with within-target conditional diversity (0.6370.637), and Appendix B.8 uses 20 reference targets in Table 1 versus two in Table 2.
Questions for authors
  1. How do ΔNLL\Delta_{\mathrm{NLL}} and target purity change when replacing ESM-2 features with learned POI ID or fixed random vectors?
  2. Does Appendix B.3 keep each canonical molecule in one split and exclude validation and test molecules from pretraining?
  3. How do Table 1 metrics change under splits that separate scaffolds or chemical series?
  4. In Table 3, what is the unseen warhead rate against all training warheads pooled, and what is the Appendix B.9 decomposition survival rate alongside target purity on warheads alone?
2 comments

We thank Reviewer wPM9 for the constructive feedback. We address the comments and concerns below.

  1. We chose ESM-2 because its representations encode sequence-derived biological relationships between proteins. These representations can also be computed for unseen POIs, unlike arbitrary target identifiers, which do not encode such relationships. If an unseen protein shares relevant features with proteins encountered during training, its embedding may allow the model to transfer learned chemical preferences. This was our main motivation for using ESM-2. Following your feedback and that of reviewer fUbM, we are testing the contribution of ESM-2 by training a random-vector baseline. Each training POI receives a fixed 1,280-dimensional random vector. The model architecture and training procedure remain unchanged. Random vectors may perform as well as, or even better than, ESM-2 on seen POIs. After all, both provide a fixed representation that allows the model to learn the chemical distribution of each training target. However, random vectors only identify proteins, whereas ESM-2 also captures relationships between their sequences. The main question is whether these relationships help TACFormer generalize to unseen POIs. We are also investigating residue-level cross-attention as an extension of this question. Mean pooling compresses residue-level information into one vector and may obscure sequence regions important for protein recognition. This may be particularly limiting to zero-shot generation. New experiments are trained on a scaffold-disjoint data split. Results from these experiments are not yet available.

  2. We deduplicated molecule–POI pairs, but the same molecule can appear in different partitions under different POIs. We have since enforced molecule-disjoint partitions and retrained the TACFormer. The new results are consistent with the submitted results.

  3. We will report results on the scaffold-disjoint split (currently running) in the revised manuscript, and under the new split, there are no test molecules that overlap with pretraining data. There was 1.70% overlap with pretraining in the submitted version.

  4. The 89.18% warhead novelty reported in the submission was calculated separately for each POI, using its training warheads as the reference. We are extending this analysis to measure novelty against the pooled training warhead repertoire as suggested. We also clarified in the manuscript that the purpose of our warhead analysis is to assess structural exploration at the POI-binding end. As discussed in Section 6 and Appendix D.1, warhead novelty alone does not establish target relevance and should be interpreted alongside whole-molecule conditional-fidelity measures. PROTAC-Splitter successfully decomposes 99.99% of unique valid generated PROTACs, and parent-warhead processing succeeds for 99.97%. Finally, we clarify the diversity comparison. The unconditional diversity reported in the submission (0.790) is calculated globally across generated molecules, whereas conditional diversity (0.637) is calculated separately for each POI and then averaged. These values therefore describe different aspects of the generated distributions and should not be interpreted as a direct comparison of diversity. We will clarify the above points in the revised manuscript.

I am still not convinced that ESM-2 is necessary for the proposed conditioning mechanism. The new comparison with unconditional generation shows that the model learns target-dependent chemical distributions, but this was not really my concern. As I understand it, the same behavior could be achieved using a learned target ID or a fixed random vector, since each POI is represented by a single fixed embedding during training. The authors also acknowledge this possibility. I agree that ESM-2 could potentially help with zero-shot generalization, but the current zero-shot results do not support this claim. I think the random-vector baseline is important for establishing what the protein representation actually contributes.

Reviewer_11ScOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors have proposed a transformer-based generative model for targeted protein degradation via PROTAC design generation.

Critical strengths
  1. The topic on targeted protein degradation is a trendy topic in modern drug discovery.
  2. The authors have identified a legitimate gap on protein target conditioning for PROTAC design.
Critical weaknesses
  1. The study is lack of a clear architecture illustration on the transformer-based generative modelling for ICLR.
  2. From the computational point of view, adding a transformer GPT decoder behind ESM-2 is not very impressive nor novel.
  3. From the biological point of view, the unseen POI performance on two POIs in Table 2 may be too limited. In addition, data redundancy / overlapping issue is speculative and not clear enough.
  4. From the technical point of view, the proposed approach has not been compared with alternative approaches. Parameter analysis and ablation analysis are highly desired in ICLR, although it may not be important in my view.
Questions for authors

See the weaknesses.

2 comments

We thank Reviewer 11Sc for the constructive feedback. We address their comments and concerns below.

  1. We agree that the architecture illustration should be included. We will include this in the revised manuscript.

  2. We emphasize that novelty should be considered in relation to the research objective and the methodological choices made to realistically achieve it. Our main contribution is not a novel transformer architecture, but rather a framework for direct protein-conditioned generation of complete PROTAC molecules under limited data, from data curation to the evaluation design. We kindly refer Reviewer 11Sc to our response to Reviewer fUbM for a more detailed discussion of our architectural choices.

3 and 4. We acknowledge that evaluating only two unseen POIs is insufficient to establish generalization. We have therefore extended our zero-shot evaluation to four POIs with different levels of similarity to the training proteins. Regarding data overlap and additional comparisons and ablations, we kindly refer the Reviewer 11Sc to our responses to Reviewers fUbM and wPM9 for further details.

In the context of ICLR which is to learn representations, I cannot increase the score anymore although the authors have responded to my comments.

Reviewer_fUbMOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TACFormer generates complete PROTAC molecules—compounds intended to bring a target protein and a degradation-related enzyme together—using the target protein’s sequence as input. It combines frozen ESM-2 protein embeddings with a SMILES transformer, first pretrained on 79,406 PROTAC-like molecules and then fine-tuned on reported PROTACs. Across 20 targets seen during training, it generates mostly valid, training-novel structures that resemble known chemistry associated with the requested target. However, conditioning does not transfer reliably to the two unseen targets tested, and the study does not assess whether generated molecules actually bind or degrade their intended targets.

Critical strengths

Generating the whole PROTAC rather than only its linker is a worthwhile objective, and direct protein conditioning provides a convenient interface for proposing molecules. The data integration and target-annotation normalization are useful contributions in a field with limited, fragmented data. Evaluation goes beyond SMILES validity: it measures likelihood preference for the correct target, proximity to held-out target-associated chemistry, and diversity of predicted warheads. The reported 98.30% validity, 91.59% novelty among unique valid molecules, and 82.11% five-neighbor target purity support the narrower conclusion that the model learns distinct chemical distributions for familiar targets. The authors also deserve credit for clearly reporting failed zero-shot transfer and acknowledging that chemical similarity does not establish biological activity.

Critical weaknesses

The architecture largely combines established components, and the experiments do not show why ESM-2 representations are necessary: learned target-ID embeddings or fixed random target vectors might produce similar results on seen targets. There are no direct comparisons with competing PROTAC generators and few ablations isolating pretraining or the conditioning mechanism. Evaluation relies on random molecule–target association splits rather than scaffold- or chemical-series-disjoint splits; moreover, the description does not clearly establish whether a multi-target molecule is kept in one partition across all its associations. This leaves uncertainty about generalization beyond familiar chemistry.

Questions for authors

see above

1 comment

We thank Reviewer fUbM for the constructive feedback. We address their comments and concerns below.

  1. We agree that TACFormer builds on established architectural components. Our design choices were motivated by three main challenges in PROTAC data: the scarcity of available molecules, fragmented datasets with inconsistent POI annotations, and a highly imbalanced distribution across targets. After normalization of POI annotations, we get 408 POIs; 94 have only one molecule, 168 have fewer than five, and 230 have fewer than ten. Even among the 20 most represented POIs, molecule counts range from 4,933 for AR to 230 for SMARCA4. This data scarcity makes direct protein-conditioned generation challenging, given that generative models are data-hungry. We therefore adopted a SMILES-based autoregressive transformer. Randomized SMILES provide alternative representations of the same molecule, enabling data augmentation without requiring additional molecules (Arús-Pous et al., 2019). Pretraining on larger molecular datasets followed by fine-tuning on smaller task-specific datasets is an established approach in molecular generation (Segler et al., 2018; Moret et al., 2020).

  2. We chose ESM-2 because its representations encode sequence-derived biological relationships between proteins. These representations can also be computed for unseen POIs, unlike arbitrary target identifiers, which do not encode such relationships. If an unseen protein shares relevant features with proteins encountered during training, its embedding may allow the model to transfer learned chemical preferences. This was our main motivation for using ESM-2. We acknowledge, however, that a comparison with fixed random protein vectors, for instance, is necessary to determine whether ESM-2 provides an advantage beyond protein identification.

  3. Our initial dataset was deduplicated at the molecule–POI pair level because the same PROTAC can target multiple POIs. However, the same molecule could still appear in different splits. Our overlap analysis identified 59 test and 46 validation molecules that also appeared in training. We therefore built a molecule-disjoint split by assigning each standardized molecule and all its POIs to the same partition, and retrained TACFormer. The results remain consistent with our submitted findings. We are also retraining TACFormer on a stricter scaffold-disjoint split, with no molecular or scaffold overlap across training, validation, and test sets, including scaffold novelty as an additional metric. These experiments are ongoing, and results are not yet available. We plan to report the scaffold-disjoint results in the revised manuscript.

  4. To address the missing ablations, we introduced a matched unconditional baseline to determine whether POI chemical enrichment actually arises from protein conditioning. TACFormer achieves 84.73% mean 5-NN purity, compared with 5.02% for unconditional generation on the molecule-disjoint split. This provides evidence that protein conditioning guides generation toward POI-enriched chemical neighborhoods beyond the general PROTAC distribution. We are also training fixed random vector control to test whether ESM-2 provides an advantage beyond POI identification. In parallel, we compare mean-pooled ESM-2 conditioning with residue-level cross-attention to investigate whether retaining residue-level information improves conditional fidelity and generalization to unseen POIs. These experiments are ongoing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.