acceptodds
Under review as a conference paper at ICLR 2027

Molecules Meet Language: Confound-Aware Representation Learning and Chemical Property Steering in Transformer-VAE Latent Spaces

Abstract

Molecular generative models often assume meaningful latent geometry, but apparent property predictability can reflect sequence-level shortcuts rather than chemical organization. We study this issue in an unsupervised autoregressive Transformer-VAE trained on SELFIES. After training, we freeze the model, fit linear probes to RDKit descriptors, and use the probe weights as candidate global steering directions. To separate chemical signal from SELFIES artifacts, we introduce a confound-aware evaluation based on residualization, confound-direction alignment analysis, and decoded-molecule traversal. This is necessary because SELFIES length, branch tokens, ring tokens, and token entropy are strongly encoded in the latent space. Under this confound-aware evaluation, we find robust monotonic steering for cLogP, FractionCSP3, HeavyAtomCount, TPSA, BertzCT, and HBA. Nonlinear probes further show that some properties admit stable global directions, while others are better described by local latent gradients. Overall, our results show that chemically meaningful steering can emerge in entangled molecular latent spaces, but only when validated through decoded molecules and controlled for representation-level confounds.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, Variational Auto Encoders, Transformers, Generative Models, AI for chemistry

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ohEUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies property encoding and steering in SELFIES-based Transformer VAEs. It compares autoregressive and non-autoregressive variants using linear and MLP probes, examines sequence-level confounds through residualization and directional alignment, and evaluates probe-derived directions by decoding latent traversals and recomputing RDKit descriptors.

Critical strengths
  1. The paper addresses a relevant question: whether property information in molecular representations translates into control over decoded molecules.

  2. Steering is evaluated using recomputed molecular properties, including raw and residual directions across 50 held-out seeds.

  3. Cross-fitted nonlinear nuisance models and held-out residual checks provide useful controls for the measured SELFIES confounds.

Critical weaknesses

1. Figure 1 contains substantive errors and is difficult to read: In SELFIES 2.1.1, [C][C][=O][O] decodes to acetaldehyde (CC=O), not the acetic acid (CC(=O)O) shown in the figure. The aromatic example also uses the unsupported [c] token. The direct connection from the latent vector to “SELFIES decoding” conflates the neural decoder with the string parser. Crowded text, small labels, and inconsistent arrow alignment and connection points further obscure the pipeline. As the main methodological overview, this figure requires both a correctness check and substantial redesign.

2. Table 20 is inconsistent with the reported probe results: All 30 AR ΔR2\Delta R^2 entries in Table 20 disagree with the corresponding differences computed from Tables 3, 4, and 19. For example, the reported values give raw gaps of 0.0869 for FractionCSP3 and 0.1401 for TPSA, versus 0.067 and 0.078 in Table 20. The monotonic-group means likewise become 0.0909/0.2135 for raw/residual targets, versus 0.061/0.152. These discrepancies exceed rounding error and cannot all be explained by substituting the Ridge baseline. Appendix A.1 describes several probe configurations without specifying which produced each table. The results need to be reconciled before the quantitative interpretation of these gaps can be assessed.

3. The interpretation conflates predictability, monotonicity, and steerability: Sections 3.5–3.6 infer approximate global linearity from successful monotonic traversal and use larger MLP–linear gaps to argue against stable global directions. Neither implication follows. For example, y(z)=exp⁡(w⊤z)y(z)=\exp(w^\top z) is nonlinear but strictly monotonic along the fixed direction ww. An MLP improvement therefore does not establish that local steering is necessary, and successful traversals on sampled seeds do not establish a globally affine property map. Figure 1’s implication that high raw and residual R2R^2 establishes steerability is similarly too strong. Appendix F.1 proposes gradient-based local steering but provides no decoded results to validate this interpretation.

4. The comparisons do not isolate the source of the AR model’s advantage: Both Non-AR baselines use an explicit length-prediction loss (Table 7), directly targeting a quantity treated as a confound, whereas the AR model does not. The models also differ in depth, latent dimensionality, and KL scheduling. Traversal protocols introduce further differences, including fixed versus free decoding length and standardized versus original latent coordinates. These experiments compare particular systems, but do not establish that autoregression explains their differences in latent organization or controllability. Matched evaluation protocols and targeted ablations of the length objective and model capacity are needed to support that attribution.

5. Decoded steering quality is insufficiently characterized: Table 16’s median per-seed Spearman scores support overall property ordering, but do not establish fine-grained or structurally coherent control. A 100-point path containing only three repeated molecular states can achieve ρ≈0.943\rho\approx0.943. Although Appendix E.1 lists additional diagnostics, the steering results do not adequately report path diversity, monotonicity violations, or similarity to the seed. The nonlinear nuisance check is useful, but addresses only the measured confounds; reduced directional alignment also does not establish reduced confound variation in decoded molecules. Per-path statistics, representative molecular edits, decoded confound trajectories, and matched random/confound-direction controls are needed to substantiate the stronger steering claims.

Questions for authors

Please see the Weaknesses section above.

Reviewer_nhe1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether an unsupervised SELFIES-based Transformer-VAE learns chemically meaningful and steerable latent representations. The authors fit post-hoc linear and nonlinear probes to predict RDKit molecular descriptors and introduce a confound-aware evaluation based on residualization against SELFIES-level statistics. They further evaluate property steering through decoded molecular traversals.

Critical strengths

The paper attempts to distinguish molecular property predictability from representation-level shortcuts and evaluates latent steering using decoded molecular properties rather than relying solely on probe predictions.

Critical weaknesses
  1. Limited novelty. The core framework largely combines existing components, including Transformer-VAEs, linear probing, residualization, and latent traversal. The methodological contribution beyond these established techniques is unclear.
  2. Insufficient baselines. The experiments mainly compare three internal VAE variants, without competitive molecular steering methods. Moreover, the variants differ in multiple architectural components, making the comparisons difficult to interpret.
  3. Incomplete confound control. The analysis considers only four SELFIES-level statistics. Important factors such as molecular composition, scaffold identity, and structural motifs remain uncontrolled, weakening the claims of chemically meaningful latent organization.
  4. Questionable traversal evaluation. The traversal range of [-150, 150] is unusually large, yet no analysis establishes whether the traversed points remain within the learned latent distribution. The reported monotonicity may therefore reflect unrealistic latent extrapolation.
  5. Lack of chemically meaningful editing evidence. Figure 3 reveals substantial structural discontinuities during interpolation. The steering experiments also lack systematic evaluation of scaffold preservation, structural similarity, and synthetic feasibility, making property monotonicity alone insufficient.
  6. Insufficient validation of nonlinear steering. The nonlinear analysis primarily compares predictive R² values. Although local gradient-based steering is discussed in Appendix F, its effectiveness on decoded molecules is not experimentally demonstrated.
Questions for authors

Please see the weaknesses.

Reviewer_TvU2OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates whether Transformer-VAEs trained on SELFIES based molecule sequences learn chemically meaningful and steerable representations. It creates a framework that separates chemical seignals from sequence level artifacts. This aims to separate the property prediction success due to capturing sequence level artificats from learning genuine representations that are useful for steering.

Critical strengths
  • The authors validate the steering capability by moving in the latent space and decoding points back into molecular graphs, instead of just property prediction.
Critical weaknesses
  • All the properties evaluated are easily calculatable RDKit properties. This creates a problem - since the properties themselves are calculated using the graph structure which is captured through SELFIES, it is possible that effect of SELFIES artifacts are not completely separated.
  • The evaluation controls for only basic SELFIES sequence statistics.
Questions for authors

Is it possible to evaluate this on complicated properties (For example, binding to a target)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.