acceptodds
Under review as a conference paper at ICLR 2027

MIP²: Information-Preserving Projector for Molecular Large Language Models

Abstract

Molecular large language models (LLMs) use cross-modal projectors to map pretrained molecular features into the language model embedding space. However, existing projectors do not explicitly enforce the recoverability of these features after projection. Therefore, the projectors may limit access to fine-grained features captured by molecular encoders. To address this challenge, we propose Molecular Information-Preserving Projector (MIP²), a molecular-language projector that combines feature preservation with flexible nonlinear adaptation. MIP² first integrates topological and geometric features through reversible coupling, preserving both encoder representations. It then separates feature preservation and nonlinear adaptation into orthogonal subspaces, yielding an injective projection with an explicit recovery map. MIP² further retains one token per atom and enables bidirectional attention within the molecular span to support atom-level structural interactions. Experiments on molecular question answering (MoleculeQA dataset), description generation and retrosynthesis (Mol-Instructions dataset), and property prediction (Mol-Instructions and eight MoleculeNet datasets) demonstrate the effectiveness of MIP². Backbone comparisons and ablation studies further support the benefits of the proposed design.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Molecular LLM, Multimodal Molecular Representation Learning, Cross-Modal Alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5Q29OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MIP², a molecular-language projector that combines reversible 2D–3D feature fusion, orthogonal information-preserving projection, and atom-level attention. The method guarantees recovery of pretrained molecular encoder features and reports improvements across several molecular tasks.

Critical strengths
  1. The proposed projector provides a clear mathematical guarantee of feature recoverability, supported by numerical verification.

  2. The experiments cover multiple downstream tasks and LLM backbones, with useful ablation studies.

Critical weaknesses
  1. The importance of exact recoverability is insufficiently justified. The paper demonstrates that encoder features can be recovered, but does not clearly explain why this property is necessary or beneficial for downstream tasks. Non-recoverable embeddings may still preserve task-relevant information and perform equally well.

  2. The performance gains are not convincingly attributed to recoverability. Removing the orthogonal constraint changes both recoverability and the representation architecture. Thus, the improvements could arise from architectural regularization rather than information preservation itself.

  3. Baseline comparisons are not fully controlled. MIP² has more parameters than the MLP baseline, while Q-Former uses a different tokenization strategy. Several comparisons also rely on previously reported results under different experimental settings.

Questions for authors
  1. Can the authors demonstrate that existing projectors actually lose task-relevant molecular information, and that such information loss negatively affects downstream performance?

  2. Can the authors isolate the benefit of exact recoverability from other architectural factors through more controlled, parameter-matched comparisons?

3 comments

Thank you for recognizing the mathematical guarantee and experimental coverage. Hope our response could address your concerns.

W1 and Q1: Task-relevant information retention and downstream utility

The submission already includes a controlled intervention demonstrating downstream use of the preserved channel. Specifically, Section 3.5, in the paragraph “Inference-time intervention on the two branches,” and Fig. 2(c) report the effect of removing the preserved branch while keeping all trained weights fixed and matching molecular embedding RMS. MoleculeQA accuracy drops from 71.28% to 58.76%. This intervention demonstrates that the trained model uses the preserved information for molecular prediction, extending the evidence beyond numerical feature recovery.

To directly examine existing projectors, we additionally train independent feature decoders and linear probes on their frozen representations, using held-out molecules and matched decoder capacity and training budgets. All reconstruction results below use independently trained decoders, providing a common evaluation protocol across projectors.

Projector2D reconstruction NMSE ↓3D reconstruction NMSE ↓Linear probe: functional-group F1 (%) ↑Linear probe: ring-count MAE ↓
MLP0.078710.2203967.960.52414
Released fixed-query Q-Former0.194280.2186162.330.59091
MIP²0.032600.0326673.210.48371

MIP² achieves lower reconstruction error for both modalities and stronger structural probe performance. Relative to MLP, it improves functional-group F1 by 5.25 percentage points, demonstrating improved accessibility of task-relevant molecular content under the same readout budget.

W2, W3, and Q2: Controlling architecture, capacity, and tokenization

Our constraint ablation changes only the complementary-subspace projection of the residual, with no change to the trainable architecture or parameter count:

(I−QQ⊤)Gϕ(u)  ⟶  Gϕ(u).(I-QQ^\top)G_\phi(u)\;\longrightarrow\;G_\phi(u).

Both variants retain the same reversible fusion, QQ, GϕG_\phi, dimensions, learned scales, QR updates, tokenization, attention, and training protocol. The comparison therefore isolates enforcement of the complementary-subspace constraint. Specifically, Q⊤r(u)=0Q^\top r(u)=0 prevents nonlinear adaptation from overwriting the anchor coordinates.

We further match the MLP’s total projector parameter budget, including fusion, while retaining the same fusion architecture, atom-level tokenization, attention, and training configuration. We also evaluate an N-token Q-Former to match MIP²’s molecular token count.

ProjectorParams (M)TokensTotal accuracy (%) ↑Macro accuracy (%) ↑
Original atom-level MLP7.98NN69.73 ± 0.6363.04 ± 0.79
Parameter-matched MLP11.96NN69.59 ± 0.8162.95 ± 0.78
Unconstrained MIP²11.96NN70.41 ± 0.6163.41 ± 0.36
MIP²11.96NN71.14 ± 0.3664.08 ± 0.69
Released fixed-query Q-Former150.93867.66 ± 0.9161.68 ± 0.83
N-token Q-Former151.38NN66.89 ± 0.7661.14 ± 0.72

MIP² exceeds the parameter-matched MLP by 1.55 percentage points and the N-token Q-Former by 4.25 points in mean total accuracy. Its advantage therefore persists under matched projector capacity and matched molecular token count.

The N-token Q-Former uses a trainable 600×768600\times768 query bank, activating the first N queries for a molecule with N heavy atoms. Its Q-Former, blending, LayerNorm, and output-projection architectures are retained, with their weights initialized directly from the released checkpoint. The first 8 queries are copied from the pretrained queries, additional queries are initialized by cycling these vectors with small, seeded Gaussian perturbations. All query embeddings are trainable, query padding is masked, and only valid outputs are passed to the LLM. This variant reuses the pretrained interface with an expanded query bank. We also retain the original eight-query checkpoint configuration as the established baseline.

Published baseline scores are explicitly identified as reference results, with their sources and original training regimes documented. Our projector-specific conclusions are supported by comparisons rerun under a unified training and evaluation protocol.

We respectfully hope you could increase your overall rating if our response have addressed your concerns.

Thank you for your detailed rebuttal. Your responses have addressed my concerns, and I have updated my score to 3.

AuthorsOpenReview ↗

Thank you.

Thanks for your follow-up. We appreciate your insightful comments. Clarifying the importance of exact recoverability is essential for justifying our motivation. We will incorporate the additional results into the camera-ready version. Again, thank you for your constructive feedback.

Reviewer_rsgGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an information-preserving projector for molecular LLMs. It combines reversible 2D–3D fusion with an orthogonal projection and a complementary nonlinear residual, allowing exact recovery of encoder features. Atom-level tokens and bidirectional molecular attention support downstream molecular tasks.

Critical strengths
  1. The projector design is clear and mathematically sound. The explicit inverse provides a concrete feature-preservation guarantee, while the complementary residual allows nonlinear adaptation.

  2. The controlled experiments comparisons provide useful evidence.

Critical weaknesses
  1. The mechanism analysis verifies recoverability more clearly than it explains the performance gains. As it acknowledges, the nominal recovery error does not establish information loss in the unconstrained baseline. The minor improvement in Table 7 supports the constraint, but does not distinguish information preservation from other optimization effects. A more direct comparison of retained information would strengthen the explanation.

  2. The paper would benefit from an additional ablation on description generation or property prediction: replace MIP² with a standard MLP while keeping other settings unchanged. This would help confirm how much the proposed module contributes to the improvements on these tasks.

Questions for authors

See weeknesses.

2 comments

Thank you for your careful reading and helpful comments. We address both concerns below.

W1: Feature recovery and task-relevant structural readouts

We have added a targeted comparison of learned feature recovery and molecular-structure readouts. In particular, we include the unconstrained MIP² variant to examine the complementary-subspace constraint under the same trainable architecture and parameter budget. Each frozen projector is paired with a separately trained decoder, with decoder capacity and optimization budgets matched across methods. All reconstruction scores are evaluated on held-out molecules using these learned decoders.

Projector2D reconstruction NMSE ↓3D reconstruction NMSE ↓Linear probe: functional-group F1 (%) ↑Linear probe: ring-count MAE ↓
MLP0.078710.2203967.960.52414
Released fixed-query Q-Former0.194280.2186162.330.59091
Unconstrained MIP²0.042330.0560070.390.49809
MIP²0.032600.0326673.210.48371

The constrained/unconstrained comparison directly addresses the mechanism question: enforcing the constraint lowers reconstruction error for both modalities, improves functional-group F1 by 2.82 percentage points, and reduces ring-count MAE from 0.49809 to 0.48371. These measurements connect the constraint to improved feature decodability and structural readouts while holding network architecture and capacity fixed.

The submission also provides a complementary behavioral test in Section 3.5, “Inference-time intervention on the two branches,” and Fig. 2(c). With trained weights fixed and molecular embedding RMS matched, deleting the preserved branch causes a 12.52-point accuracy decline, from 71.28% to 58.76%. Together, the learned-readout comparison and this intervention establish that the preserved representation is accessible to downstream readouts and actively used in molecular prediction.

W2: Controlled ablation on description generation

We add the requested MLP replacement on both description generation and numerical property prediction. Within each task, the molecular encoders, SELFIES/graph/3D inputs, atom-level tokenization, bidirectional molecular attention, LoRA configuration, training schedule, and decoding protocol remain unchanged.

ProjectorBLEU-2 ↑BLEU-4 ↑ROUGE-1 ↑ROUGE-2 ↑ROUGE-L ↑METEOR ↑Property MAE ↓
Standard MLP0.3850.3440.7290.6670.7210.6550.003961
MIP²0.3990.3540.7440.6790.7350.6640.003529

With these concerns addressed, we sincerely hope you could increase your rating.

AuthorsOpenReview ↗

Regarding Our Rebuttal

Thank you again for your time and constructive feedback on our manuscript. We have carefully addressed concerns you raised above. We would greatly appreciate it if you could take another look at our responses. Your feedback would be very helpful in ensuring that we have adequately addressed your comments before the discussion period concludes. We are glad to engage in further disscussion or provide additional clarifications if needed.

Reviewer_BYDVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MIP², a molecule-to-language projector designed so that frozen 2D and 3D encoder features remain exactly recoverable after projection into the LLM embedding space. It fuses the two modalities with a reversible additive coupling, embeds the result through a learned matrix Q with orthonormal columns, and restricts a nonlinear residual to the orthogonal complement of Col(Q), giving an explicit left inverse. Each heavy atom is represented as one token, with bidirectional attention within the molecular span. The authors evaluate on MoleculeQA, three Mol-Instructions subsets, and eight MoleculeNet datasets, and it gains over MLP and Q-Former projectors in controlled comparisons.

Critical strengths

Principled information preservation and empirical gains. The paper makes feature preservation an architectural constraint: fused features occupy a learned subspace Col(Q), and the nonlinear residual is confined to its orthogonal complement, giving a closed-form left inverse for any G (Prop. 1). The guarantee holds in practice (Fig. 2a) at near-MLP inference cost (Table 16), and under matched settings the projector outperforms MLP and Q-Former baselines on MoleculeQA (Table 6) and retrosynthesis (Table 12). This simple decoupling of preservation and adaptation is the paper’s core contribution and is easy to adopt in other projectors.

Critical weaknesses

The contribution of information preservation to performance is not demonstrated. The paper attributes its gains to preserving encoder information, but its experiments vary architectural components, not the amount of information preserved. The near-zero recovery error (Fig. 2a) confirms a property that holds by construction rather than its benefit, and apart from two examples in Fig. 3, the paper does not measure whether the baseline projectors lose encoder information. The authors also note that the unconstrained variant’s failed recovery does not imply information loss (App. F.3). Moreover, although preservation does not require 2D–3D fusion, the projectors are compared only with both modalities (Tables 5, 6, 12). Whether the gains stem from preservation, the paper’s central claim, therefore remains unshown.

Comparisons with prior works. EDT-Former, the main baseline in Tables 1–3, is pretrained on Mol-LLaMA-Instruct, and the baselines it re-ran follow its fine-tuning protocol. In contrast, the proposed model has no pretraining stage and uses a different training schedule, and the paper does not discuss how these differences affect the margins. Table 4 also mixes specialist and generalist regimes. Among its baselines, only MR-MoL is trained on the same split and the other generalists are released checkpoints. The split-matched specialist results reported by MR-MoL, against which the proposed model is best only on HIV and SIDER, are omitted. Without aligned budgets or a consistent grouping of specialists and generalists, the margins over prior work are hard to interpret.

Questions for authors

Clarifications

  • Does the MLP baseline (App. C.5) receive the coupled features u or the raw concatenation? If the latter, the gap in Table 6 mixes the effects of the coupling and the constrained projection.
  • Which projector is used for the 2D-only and 3D-only rows of Table 11, and could the MLP be compared on the same inputs?
  • Is there direct evidence that preservation drives the gains, for example by probing whether the atom-level MLP loses encoder information, or by measuring accuracy as less information is kept in the anchor?
  • Does the advantage persist under the same training budget as a strong baseline (EDT-Former or MR-MoL), when the proposed model adds an alignment pretraining stage, or when it is trained as a single model across all benchmarks? Could the split-matched specialist results reported by MR-MoL be added to Table 4?
  • On description generation, the proposed model has lower BLEU than the strongest baselines but the highest ROUGE and METEOR (Table 2). Could you explain what causes this discrepancy?

Suggestions for future work

  • Ablating a frozen versus learned Q, per-atom versus pooled tokens, a parameter-matched MLP, and a non-invertible fusion of the same size would support design choices that currently rest on argument.
  • Some baseline labels copied from EDT-Former appear swapped. LLaMo is listed as Text although it uses graph inputs, and Mol-Inst.-Llama3.1 as T/G although it is a text-only model. These should be verified against the original papers.
5 comments

Thank you for your helpful comments. Hope our response could address your concerns.

W1 and Q1–Q3: Feature preservation, fusion, and modality controls

We add direct measurements of feature recovery and task-relevant structural content. Each frozen projector is evaluated with its own independently trained decoder, using held-out molecules and matched decoder capacity and training budgets. All reconstruction scores below therefore come from learned decoders, including those for the unconstrained variant.

Projector2D reconstruction NMSE ↓3D reconstruction NMSE ↓Functional-group linear-probe F1 (%) ↑Ring-count linear-probe MAE ↓
MLP0.078710.2203967.960.52414
Q-Former0.194280.2186162.330.59091
Unconstrained MIP²0.042330.0560070.390.49809
MIP²0.032600.0326673.210.48371

Relative to the unconstrained variant, MIP² improves functional-group F1 and reduces reconstruction error in both modalities. These comparisons directly examine empirical feature recovery and structural readouts under the same trainable architecture and parameter budget.

The submission also already evaluates downstream use of the preserved channel in Section 3.5, “Inference-time intervention on the two branches,” and Fig. 2(c). Removing this branch at fixed weights, with molecular embedding RMS matched, reduces accuracy from 71.28% to 58.76%.

We clarify the original input configurations explicitly:

Implementation detailConfiguration
Input to the original MLP in App. C.5Coupled features u
Projector for the original 2D-only row in Table 11MIP$^2$
Projector for the original 3D-only row in Table 11MIP$^2$

To separate projection from fusion, the added parameter-matched MLP uses the same reversible fusion architecture as MIP². We additionally compare the projectors under single-modality inputs.

InputFusionProjectorMoleculeQA total accuracy (%) ↑Macro accuracy (%) ↑
2D onlyNoneParameter-matched MLP68.2361.37
2D onlyNoneMIP²69.3962.72
3D onlyNoneParameter-matched MLP67.3560.86
3D onlyNoneMIP²69.8663.56
2D + 3DReversible couplingParameter-matched MLP69.4362.44
2D + 3DReversible couplingMIP²71.2864.37

The dual-modality comparison retains a 1.85-point accuracy advantage with both fusion architecture and projector capacity controlled.

W2 and Q4: Training regimes, budgets, and specialist comparisons

The statement that Table 4 mixes specialist and generalist models is incorrect: its listed baselines are generalist molecular LLMs. Our MoleculeNet results also already come from one model trained jointly across all eight datasets, as specified in App. D.4. We will add the specialist results reported by MR-Mol to the revised version as you suggested.

To make the training differences explicit, we summarize the alignment stages, downstream schedules, trainable parameters, and computational costs below. MIP² achieves its reported results without an additional molecular–language alignment stage, using fewer trainable parameters and fewer downstream epochs than the compared baselines.

BenchmarkModelAdditional alignment dataDownstream TrainingTotal trainable parametersGPU-hours / hardwareSource
MoleculeQAMIP²None3 epochs53.91M7.05h / 1× RTX 4090Our training logs
MoleculeQAEDT-FormerDynamic Query Transformer pretraining: 10 epochs (15,599 steps); alignment tuning on Mol-LLaMA-Instruct: 284,743 instances, 2 epochs5 epochsAlignment213M Downstream 233.97MStage2 Alignment 10.28h Downstream 53.82h / 1× RTX 4090Published protocol + our reproduction
MoleculeNetMIP²None3 joint-task epochs53.91M15.5h / 1× RTX 4090Our training records
MoleculeNetMR-MoL89,919 molecule–text pairs; 20 epochs, with the epoch-10 checkpoint selected5 joint-task epochsAlignment 99.9M Downstream 111.7MAlignment 30h Downstream 30h / 1× RTX 5090Published protocol
We also add a alignment stage as suggestion and provide the results below. The alignment stage affects little to the final results.
ModelAdditional alignment dataAlignment epochsDownstream epochsTotal accuracy (%) ↑Macro accuracy (%) ↑
MIP²None0371.2864.37
MIP² + alignmentMol-LLaMA-Instruct2371.0964.16

Q5: Description-generation metrics

BLEU emphasizes modified n-gram precision and includes a brevity penalty. ROUGE measures token and subsequence overlap, while METEOR evaluates unigram alignment with an ordering penalty. Their weighting and aggregation differ, so their rankings need not coincide.

We add a length and n-gram decomposition to identify the source of the observed difference:

ModelOutput/reference length ratioBLEU brevity penaltyModified bigram precisionModified four-gram precision
MIP²0.66850.60910.61340.4849

The diagnostics reveal a substantial brevity penalty in our BLEU evaluation. The output/reference length ratio is 0.6685, yielding a brevity penalty of 0.6091. Since BLEU multiplies its geometric mean of n-gram precisions by this factor, the penalty reduces the score by approximately 39.1%. Meanwhile, modified bigram and four-gram precisions are 0.6134 and 0.4849, respectively. Output brevity therefore contributes substantially to the reported BLEU scores. ROUGE and METEOR use different overlap, alignment, and length treatments, so their rankings need not coincide with BLEU. We could release the description that generated by our model after acceptance.

Suggestions for Design ablations and modality metadata

The parameter-matched MLP comparison above addresses the capacity suggestion. We additionally evaluate frozen versus learned QQ, pooled molecular tokens, and non-invertible fusion with matched fusion dimensions and parameter count.

VariantMoleculeQA total accuracy (%) ↑Macro accuracy (%) ↑
MIP² with learned QQ71.2864.37
MIP² with frozen orthonormal QQ69.3162.29
MIP² with pooled molecular tokens66.4560.68
MIP² with parameter-matched noninvertible fusion70.4663.87
Parameter-matched MLP69.4362.44

We will correct the modality metadata to T/G for LLaMo and T for the text-only Mol-Instructions baseline in the revised version, thanks for your kind remention.

Please consider raising your rating if our response addressed your concern.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.