acceptodds
Under review as a conference paper at ICLR 2027

Capability-Preserving Molecular Adaptation via Reciprocal Preference Optimization

Abstract

Molecular large language models (LLMs) are commonly adapted from general-purpose LLMs through large-scale domain-specific supervised fine-tuning (SFT). While effective at improving targeted molecular tasks, such specialization may overwrite capabilities already present in the base model, leading to degradation in general-purpose abilities and molecular reasoning capabilities outside the fine-tuning distribution. In this work, we investigate whether preference optimization can provide a more capability-preserving alternative for molecular adaptation. Our key hypothesis is that, by learning relative preferences rather than directly maximizing the likelihood of target responses, while remaining anchored to a reference policy, preference optimization can induce a more conservative adaptation regime than conventional SFT. To make preference optimization feasible without human-annotated molecular preference data, we introduce ReMPA, a Reciprocal Molecular Preference Adaptation framework. ReMPA repurposes existing paired molecule-response supervision into reciprocal preference pairs by pairing template-aligned, structurally similar molecules, such that each response is preferred under its corresponding molecular condition and rejected under the other. Across molecular property prediction, molecular reasoning and comprehension, and evaluations of general capabilities, our experiments demonstrate that, compared with supervised fine-tuning, ReMPA achieves meaningful gains in targeted molecular capabilities while better preserving capabilities already present in the base model. These results suggest that reciprocal preference optimization offers a promising capability-preserving alternative to conventional supervised domain adaptation for molecular LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
molecular large language models, preference optimization, domain adaptation, capability preservation, supervised fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_imWUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReMPA, a reciprocal preference optimization framework for molecular LLM adaptation. It constructs preference pairs from structurally similar molecules sharing the same instruction template, enabling domain adaptation without human preference annotations. Experiments on molecular comprehension, reasoning, property prediction, and general-purpose benchmarks suggest that ReMPA improves molecular capabilities while better preserving the base model's existing capabilities than conventional supervised fine-tuning (SFT). The problem is meaningful and the results are promising. However, concerns remain regarding the independent contribution of reciprocal preference construction, the reliability of pseudo-negative responses, and the fairness and robustness of the experimental comparisons.

Critical strengths
  1. Meaningful motivation. The paper addresses the important trade-off between acquiring domain-specific molecular capabilities and preserving the existing knowledge and reasoning abilities of general-purpose LLMs.
  2. Practical preference construction. ReMPA converts existing molecule–response supervision into reciprocal preference pairs without requiring additional human annotations or response generation.
  3. Comprehensive evaluation. The experiments cover multiple molecular and general-purpose benchmarks, with encouraging results demonstrating improved molecular performance and reduced capability degradation relative to SFT.
Critical weaknesses

W1. The contribution of reciprocal preference construction is insufficiently isolated. Although ReMPA outperforms One-way DPO on MoleculeQA and FGBench, their results on MMLU and molecular property prediction are nearly identical (Table 6). This raises the question of whether the primary benefit comes from reciprocal preference construction or preference optimization itself. More controlled comparisons with alternative preference construction strategies, matched training steps, and consistent checkpoint selection are needed.

W2. The reliability of pseudo-negative molecular responses is not sufficiently validated. ReMPA treats responses from structurally similar molecules as rejected alternatives. However, similar molecules may share properties, functional groups, and valid textual descriptions, making some pseudo-negative responses partially or entirely correct. The authors should quantify preference-label reliability and investigate how molecular similarity affects the quality of constructed preference pairs.

W3. The theoretical analysis does not directly support capability preservation. Appendix A shows that reciprocal preference optimization discourages molecule-independent response reweighting. However, this does not establish that the method preserves general-purpose or previously acquired molecular capabilities. Additional analysis of parameter drift, policy divergence, or prediction changes relative to the base model would strengthen the theoretical motivation.

W4. The SFT comparisons raise concerns about evaluation fairness. In Table 5, SFT-Full achieves an initial parse rate of only 5.96%, compared with 94.87% for ReMPA. Performance improves substantially after format repair, suggesting that output-format failures contribute significantly to the apparent degradation. Although the authors include a learning-rate-matched SFT control, stronger format-consistent and compute-matched comparisons are needed to isolate the benefits of ReMPA.

W5. The adaptation-preservation trade-off and statistical robustness require further evaluation. The main conclusions rely on selected training configurations, while differences between ReMPA and One-way DPO are relatively small on several benchmarks. Evaluating acquisition–preservation trade-off curves across training checkpoints and reporting variability across independent training runs would provide stronger evidence for the claimed advantages.

Questions for authors

Questions for the Authors

  1. What benefits does reciprocal preference construction provide beyond standard DPO under matched training conditions?
  2. Can the authors establish a clearer connection between the theoretical analysis and capability preservation?
  3. To what extent is the performance degradation of SFT caused by output-format failures rather than actual capability loss?
  4. Are the reported improvements consistent across multiple training seeds and different adaptation strengths?
8 comments
AuthorsOpenReview ↗

Response to Reviewer imWU(1/6): W1 & Q1

We thank the reviewer for recognizing the significance of our motivation, the practicality of our approach, and the comprehensive evaluation, and for the constructive suggestions. Below, we address key points including the contribution and reliability of reciprocal preferences, their connection to capability preservation, and evaluation fairness.

W1 & Q1. Controlled Comparison of Preference Construction Strategies

We thank the reviewer for highlighting the need to better isolate the contribution of reciprocal preference construction. To provide a more controlled comparison, we conducted an additional Non-reciprocal DPO-50K experiment that preserves the full positive-record coverage of ReMPA while removing reciprocal supervision.

Specifically, we preserved all 50,000 original training condition–response records, including their system prompts, molecular conditions, instruction templates, and chosen responses, and changed only the rejected responses. Each chosen record appears exactly once. Donors are selected from the original records within the same template and split, with a different canonical molecule. Self-pairing and reciprocal two-cycles are prohibited across the entire dataset. Thus, an assignment a → b excludes the reverse assignment b → a throughout training.

To maintain structurally plausible negatives, we maximized total Morgan-fingerprint Dice similarity under these constraints and the training-length requirements. The resulting non-reciprocal control achieves a mean training Dice similarity of 0.6585, compared with 0.7379 for ReMPA. Both variants use the same backbone, LoRA configuration, batch settings, and optimizer settings. Reference log-probabilities were recomputed for the new preferences. Our validation-based checkpoint-selection procedure selected the non-reciprocal checkpoint at step 1,955, close to the submitted ReMPA checkpoint at step 1,939.

The following table compares the new control with the submitted Table 6 results. All values are percentages. MoleculeQA accuracy is averaged over support-set seeds 0, 42, and 123, while Property BA is macro-averaged across five tasks.

ModelMoleculeQA Acc.↑FGBench Bool Avg.↑MMLU Micro Acc.↑Property Initial BA↑Property Repair BA↑
One-way DPO39.8348.6768.2753.9956.30
Non-reciprocal DPO-50K40.5246.3768.4048.5552.46
ReMPA41.2351.1068.2954.0956.27

The non-reciprocal control improves MoleculeQA accuracy over One-way DPO (40.52% vs. 39.83%). ReMPA further outperforms the coverage-matched non-reciprocal control by 0.71 percentage points on MoleculeQA, 4.73 points on FGBench, and 5.54/3.81 points on initial/repaired property BA, while MMLU remains comparable across all three models.

These results extend our original One-way DPO ablation with a stronger control for positive-record coverage and closely matched training updates. ReMPA achieves consistently stronger molecular performance than the similarity-optimized non-reciprocal control, while maintaining comparable general-capability performance. This comparison provides additional empirical evidence for the value of reciprocal preference construction in molecular adaptation.

AuthorsOpenReview ↗

Response to Reviewer imWU(2/6): W2

W2. Reliability of Constructed Molecular Preference Labels

We conducted a three-stage empirical audit combining chemical entity verification, LLM-based semantic evaluation, and independent expert assessment to quantify preference-label reliability and examine its relationship with molecular similarity (Appendix B.3, Fig. 3).

1. Chemical entity extraction and structural verification. GLiNER2 extracts chemical entities from the original responses. An LLM agent generates or reuses entity-to-SMARTS mappings, while RDKit checks their structural validity and compatibility with the source molecules. Together, the LLM agent and RDKit support the revision or filtering of non-structural, contextually irrelevant, or problematic entities. The resulting entity–SMARTS table and source-matched substructure annotations provide auxiliary evidence for semantic evaluation.

2. LLM-based semantic preference audit. We sampled 500 complete reciprocal pairs (1,000 directional preferences), balanced across instruction templates, and evaluated both directions separately. For each direction, the LLM judge receives the instruction, target SMILES, two anonymized responses, and auxiliary entity/SMARTS annotations. It compares structural accuracy, chemical and biological explanations, and instruction relevance, selecting the better response or AMBIGUOUS when no clear quality difference exists. Original preference labels are hidden, and candidate positions are balanced.

We further analyzed Morgan-fingerprint Dice similarity across judgment categories:

LLM semantic judgmentExamplesProportion (%)Mean Dice ± SDMedian Dice [P25, P75]
Chosen preferred83183.100.7264 ± 0.12470.7229 [0.6197, 0.8280]
Rejected preferred11811.800.7999 ± 0.12060.8373 [0.7098, 0.8965]
Ambiguous515.100.8330 ± 0.10790.8834 [0.7864, 0.9111]
Total1,000100.00——

The LLM judge supports 83.1% of the original preference labels, favors the rejected response in 11.8%, and identifies 5.1% as ambiguous.

Structural similarity is associated with preference-label reliability. Mean Dice similarity increases from 0.7264 for chosen-preferred cases to 0.7999 for rejected-preferred cases and 0.8330 for ambiguous cases. This trend is consistent with greater overlap in functional groups, molecular properties, and valid descriptions among highly similar molecules. These findings highlight a trade-off in similarity-based preference construction: structurally similar molecules provide plausible and challenging pseudo-negatives, while greater structural overlap can make molecule-conditioned preferences more difficult to distinguish.

3. Independent expert validation. We further selected 55 directional examples from distinct reciprocal pairs, approximately preserving the LLM judgment distribution. An independent expert evaluated anonymized responses using the instruction, target molecule's IUPAC name, and RDKit-generated 2D structure, without access to preference labels or LLM judgments.

The expert favored the originally chosen response in 51/55 cases (92.73%), the rejected response in 1/55 (1.82%), and assigned AMBIGUOUS in 3/55 (5.45%).

Expert \ LLMChosenRejectedAmbiguousTotal
Chosen455151
Rejected0101
Ambiguous1023
Total466355

Expert and LLM judgments agree on 48/55 cases (87.27%). Five of the seven disagreements involve the LLM favoring the rejected response while the expert favors the original chosen response. This suggests that the LLM-based audit may overestimate the frequency of preference reversals relative to independent expert judgments in this subset. The expert audit also shows higher mean Dice similarity for ambiguous cases (0.8970) than chosen-preferred cases (0.7118), consistent with the LLM-based trend.

Overall, the audits support the constructed preferences in 83.1% of LLM judgments and 92.73% of expert judgments, with 87.27% expert–LLM agreement. The similarity analysis further identifies high structural similarity as a factor associated with ambiguous and reversed preferences, providing empirical insight into the trade-off between challenging pseudo-negatives and preference-label reliability.

AuthorsOpenReview ↗

Response to Reviewer imWU(3/6): W3 & Q2

W3 & Q2. Connection Between Theoretical Analysis and Capability Preservation

Appendix A provides a theoretical explanation of how reciprocal preference construction shapes the adaptation process. Proposition 1 establishes that molecule-independent response reweighting cannot reduce the symmetric reciprocal loss below the reference-policy baseline. Proposition 2 further shows that improving beyond this baseline requires positive molecule–response interaction gain Γ\Gamma, while Eq. (40) demonstrates that the initial optimization gradient is directly proportional to −∇θΓ-\nabla_{\theta}\Gamma. Together, these results explain how reciprocal supervision encourages molecule-conditioned preference learning rather than globally favoring or suppressing particular responses.

Training dynamics provide additional evidence consistent with this mechanism. Our logged reference-relative likelihoods reveal a notable difference between One-way DPO and ReMPA. At their selected checkpoints, One-way DPO (step 775) exhibits a chosen log-probability change of −19.01 and a rejected change of −59.33, indicating that its preference margin improves even as the chosen-response likelihood decreases substantially. In contrast, ReMPA (step 1,939) exhibits a much smaller chosen decrease (−4.39) alongside a substantial rejected decrease (−70.69). This difference becomes more pronounced over training: at step 1,939, the One-way DPO chosen log-ratio reaches −107.37, compared with −4.39 for ReMPA, as measured on their respective validation preference sets. These observations suggest that ReMPA achieves preference separation with substantially less chosen-response likelihood displacement, rather than relying heavily on suppressing both preferred and rejected responses.

The additional Non-reciprocal DPO-50K control provides complementary evidence at the capability level. Despite comparable molecular comprehension performance (41.23% vs. 40.52% on MoleculeQA), ReMPA achieves 51.10% FGBench Boolean accuracy, compared with 46.37% for the non-reciprocal control. Relative to the Base model (52.07%), this corresponds to a degradation of only 0.97 percentage points for ReMPA, versus 5.70 points for the non-reciprocal control. This indicates substantially stronger preservation of previously acquired molecular reasoning performance under comparable adaptation gains.

Together, these findings provide a theoretical and empirical connection between reciprocal preference learning and capability preservation. The theoretical analysis shows that reciprocal supervision encourages molecule-conditioned preference discrimination rather than molecule-independent response reweighting. Consistent with this mechanism, the training logs indicate that ReMPA achieves preference separation with substantially smaller chosen-response likelihood displacement than One-way DPO, avoiding the pronounced suppression of preferred responses observed in the latter. This more conservative adaptation behavior is consistent with the substantially better retention of molecular reasoning performance observed in FGBench. These findings suggest that reciprocal preference learning can acquire molecule-specific distinctions while reducing unnecessary disruption to capabilities already present in the base model.

Reviewer_ZsvSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReMPA aims to improve molecular capabilities while preserving a language model's existing abilities. It pairs structurally similar molecules sharing an instruction template and swaps their responses to create two reciprocal preferences: each original response is preferred under its own molecular condition. Both directions are trained with standard DPO, using the original model as the reference and without intermediate molecular SFT. Experiments use Llama-3.1-8B-Instruct with LoRA and evaluate molecular comprehension, reasoning, property prediction, and general knowledge.

Critical strengths
  1. A practical way to construct preference data from existing resources. The authors propose ReMPA, which converts existing paired molecule–response data into informative preference pairs. This enables preference optimization for molecular tasks without human preference annotation or additional response generation.
  2. A systematic comparison across complementary evaluation dimensions. The authors compare the unadapted base model, conventional SFT, one-way DPO, and ReMPA on molecular property prediction, molecular reasoning and comprehension, and general capabilities. This design directly tests the paper's central hypothesis that, when applied directly to a general-purpose base model, preference optimization anchored to the reference model can acquire targeted molecular capabilities with less disruption to capabilities already encoded in the model than conventional SFT.
Critical weaknesses
  1. The constructed preferences lack semantic validation. Similar molecules can share correct descriptions, so a response's validity for its original molecule does not establish that it is worse for the paired molecule. Removing identical strings does not eliminate equivalent or jointly valid answers, and a valid preference in one direction does not guarantee the reverse. Different molecular prompts are not a formal problem for DPO; the concern is whether each conditional preference is justified. Figure 2 appropriately calls the transferred responses pseudo-negatives, but their ambiguity and error rates remain unquantified.
  2. The experiments do not isolate the benefit of reciprocity.
    • Eq. (8) averages two standard DPO losses and is equivalent, under equal weighting, to ordinary DPO on both directional examples. The contribution is therefore the data construction, not a new loss. Likewise, Appendix A explains a property of symmetric supervision rather than superiority over standard DPO.
    • The one-way control retains one direction per pair, whereas ReMPA includes both and covers more positive condition–response records. Training for 12 versus 6 epochs reasonably matches planned exposure; it does not itself establish greater forgetting. However, sample coverage and repetition differ, and the selected checkpoints are at 775 versus 1,939 steps. These choices are defensible for comparing practical recipes but leave the mechanism confounded.
    • ReMPA gains 1.40 percentage points on MoleculeQA and 2.43 on mean FGBench Boolean accuracy over one-way DPO, while MMLU and repaired property balanced accuracy remain essentially unchanged. This supports incremental molecular-task gains, not better retention attributable to reciprocity. A stronger control would preserve all positive records but sample similar donor responses without enforcing reverse pairs, matching preference-example count, similarity distribution, and actual training exposure. Training ordinary DPO on the same full reciprocal dataset would only check objective equivalence, not remove reciprocity.
Questions for authors
  1. Are both preference directions valid? Could the authors audit representative pairs to quantify valid, ambiguous, and incorrect preferences in each direction? Do the gains persist after filtering ambiguous pairs?
  2. What distinguishes ReMPA from standard DPO on both directions? Is Eq. (8) operationally equivalent to training ordinary DPO on all reciprocal examples?
  3. Does reciprocity help beyond positive coverage and training exposure? Could the authors compare against random similar-donor DPO without enforced reverse pairs, matching positive records, preference-example count, similarity distribution, and actual updates?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ZsvS(1/3): W1 & Q1

We thank the reviewer for recognizing the practicality of our preference construction and the systematic evaluation, and for the constructive suggestions. Below, we address key points including preference-label validity, the relationship between ReMPA and standard DPO, and the contribution of reciprocity under controlled training conditions.

W1 & Q1. Semantic Validity of Reciprocal Preferences

To assess whether each constructed preference is semantically justified under its specific molecular condition, we conducted a three-stage reliability audit combining chemical entity verification, LLM-based semantic judgment, and independent expert evaluation (Appendix B.3, Fig. 3).

1. Chemical entity extraction and structural verification. GLiNER2 extracts chemical entities from the original responses, preserving their textual spans and source-molecule associations. An LLM agent generates or reuses entity-to-SMARTS mappings, while RDKit checks their structural validity and matches against the source molecules. The LLM agent and RDKit checks jointly support the revision or filtering of non-structural, contextually irrelevant, or problematic entity mappings. This workflow produces an entity–SMARTS table and source-matched substructure annotations for each response, which serve as auxiliary evidence for subsequent semantic evaluation.

2. Direction-wise semantic preference validation. From the structurally annotated examples, we selected 500 complete reciprocal pairs (1,000 directional preferences), balanced across instruction templates, and evaluated both directions separately using an LLM judge. For each direction, the judge receives the instruction, target SMILES, two anonymized full-text responses, and their extracted chemical-entity/SMARTS annotations. It evaluates which response better addresses the instruction for the current target molecule, considering structural accuracy, chemical and biological explanations, and overall relevance. The original preference labels are hidden, candidate positions are balanced, and the judge selects A, B, or AMBIGUOUS when no clear quality difference exists.

The resulting direction-level judgments are:

LLM semantic judgmentExamplesProportion (%)
Chosen preferred83183.10
Rejected preferred11811.80
Ambiguous515.10
Total1,000100.00

Thus, the LLM judge supports the constructed preference ordering in 83.1% of directional examples, identifies a reversed preference in 11.8%, and finds no clear quality difference in 5.1%. These judgments explicitly consider both reciprocal directions under their respective molecular conditions, rather than inferring one direction's validity from the other.

3. Independent expert validation. To assess the reliability of the LLM judgments, we further selected 55 directional examples from 55 distinct reciprocal pairs, approximately preserving the LLM judgment distribution. An independent expert evaluated each example using the instruction, target molecule's IUPAC name, RDKit-rendered 2D molecular structure, and two anonymized candidate responses, without access to the original preference labels or LLM decisions.

The expert favored the originally chosen response in 51/55 cases (92.73%), favored the rejected response in 1/55 (1.82%), and assigned AMBIGUOUS in 3/55 (5.45%). The following three-class confusion matrix compares the expert and LLM judgments on these same examples:

Expert \ LLMChosenRejectedAmbiguousTotal
Chosen455151
Rejected0101
Ambiguous1023
Total466355

The expert and LLM judgments agree on 48/55 cases (87.27%). Among the seven disagreements, five involve the LLM favoring the rejected response while the expert favors the originally chosen response, suggesting that the LLM audit may overestimate preference reversals in this subset.

Overall, the semantic audit supports the constructed labels in 83.1% of evaluated directions, while independently obtained expert judgments favor the original chosen responses in 92.73% of the reviewed examples, with 87.27% expert–LLM agreement. These results provide direct empirical evidence for the semantic reliability of the constructed molecular preferences and quantify the proportions of supported, reversed, and ambiguous preference judgments.

AuthorsOpenReview ↗

Response to Reviewer ZsvS(2/3): Q2

Q2. Relationship Between ReMPA and Standard DPO

Given the molecule-conditioned textual preferences constructed by ReMPA, Eq. (8) is mathematically equivalent to applying standard DPO to both reciprocal directions with equal weights. The key distinction lies in the construction of preference supervision: existing molecular instruction datasets commonly provide molecule–response supervision but generally lack explicit chosen–rejected textual response pairs. ReMPA bridges this gap by pairing template-aligned, structurally similar molecules and assigning reciprocal preference roles to their original responses under different molecular conditions. This construction enables standard DPO to be applied directly to molecular textual-response adaptation without additional preference annotations or response generation. Appendix A characterizes the theoretical implications of this reciprocal supervision, while the additional non-reciprocal DPO-50K comparison in W2 & Q3 provides empirical evidence for its contribution beyond positive-record coverage.

AuthorsOpenReview ↗

Response to Reviewer ZsvS(3/3): W2 & Q3

W2 & Q3. Controlling Positive Coverage and Training Exposure

We thank the reviewer for suggesting a control that preserves positive-record coverage while removing reciprocal supervision. Our original One-way DPO comparison differs in positive-record coverage, motivating a more closely controlled comparison.

We conducted an additional Non-reciprocal DPO-50K experiment using the ReMPA dataset. We preserved all 50,000 original training condition–response records, including their system prompts, molecular conditions, instruction templates, and chosen responses, and changed only the rejected responses. Each chosen record appears exactly once. Donors are drawn exclusively from the original records within the same template and split, with a different canonical molecule. Self-pairing and reciprocal two-cycles are prohibited across the entire dataset. Thus, whenever an assignment a → b is present, b → a cannot occur elsewhere during training.

To keep negatives as structurally similar as possible, we maximized the total Morgan-fingerprint Dice similarity subject to these global constraints and the training-length requirements. This produces a similarity-optimized non-reciprocal control, with mean training Dice similarity of 0.6585, compared with 0.7379 for ReMPA. We retained the same backbone, LoRA configuration, batch settings, and optimizer settings as ReMPA, and recomputed reference log-probabilities for the new preferences. Our validation-based checkpoint-selection procedure selected epoch=04-step=00001955.ckpt (step 1,955), close to the submitted ReMPA checkpoint at step 1,939.

The following table compares this control with the submitted Table 6 results. All values are percentages. MoleculeQA reports mean initial accuracy across three 10-shot support-set seeds; FGBench reports mean Boolean accuracy across its three reasoning categories; Property reports five-task macro-averaged balanced accuracy.

ModelMoleculeQA Acc.↑FGBench Bool Avg.↑MMLU Micro Acc.↑Property Initial BA↑Property Repair BA↑
One-way DPO39.8348.6768.2753.9956.30
Non-reciprocal DPO-50K40.5246.3768.4048.5552.46
ReMPA41.2351.1068.2954.0956.27

The non-reciprocal control improves MoleculeQA accuracy over One-way DPO (40.52% vs. 39.83%). ReMPA further outperforms the coverage-matched non-reciprocal control by 0.71 percentage points on MoleculeQA, 4.73 points on FGBench, and 5.54/3.81 points on initial/repaired property BA, while MMLU remains comparable across all three models.

These results extend our original One-way DPO ablation by controlling positive-record coverage and closely matching training updates. ReMPA consistently achieves stronger molecular performance than the similarity-optimized non-reciprocal control across the evaluated molecular benchmarks, while maintaining comparable general-capability performance.

Reviewer_aKxrOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether molecular adaptation can improve domain-specific capabilities while preserving the general and molecular reasoning abilities of the original language model. The proposed framework, ReMPA, constructs reciprocal preferences from existing molecule–response supervision. Two structurally similar molecules sharing an instruction template are paired, and each original response is preferred under its corresponding molecular condition and rejected under the other. The model is adapted directly from Llama-3.1-8B-Instruct using standard DPO with the original model as a frozen reference.ReMPA improves molecular comprehension while retaining substantially more base-model performance than the main SFT baselines.

Critical strengths
  1. The preference construction is practical and has a clear theoretical motivation. ReMPA reuses existing molecule–response supervision without requiring additional human preference annotations or response generation. Assigning opposite preference roles to the same responses under different molecular conditions discourages learning a molecule-independent ranking over responses.
Critical weaknesses
  1. Sensitivity to the weighting of reciprocal losses. ReMPA assigns equal weights to the forward and reverse preference losses. Have you explored a weighted objective, αLforward+(1−α)Lreverse\alpha \mathcal{L}_{\mathrm{forward}} + (1-\alpha)\mathcal{L}_{\mathrm{reverse}}, instead of fixing α=0.5\alpha=0.5? Could unequal weighting improve the adaptation–preservation trade-off, and how would it affect the theoretical properties established for the symmetric objective?
  2. Comparison with continual learning methods. Since capability preservation is a central motivation, how does ReMPA compare with continual learning approaches such as experience replay and O-LoRA? Under comparable adaptation data and computational budgets, does ReMPA offer advantages in domain-specific performance, retention of existing capabilities, or memory and computational costs?
  3. Generalization across model families. All controlled experiments use Llama-3.1-8B-Instruct. What motivated restricting the evaluation to this backbone? Have you evaluated ReMPA on other model families, such as Qwen, to determine whether the observed adaptation–preservation benefits generalize beyond Llama?
Questions for authors
  1. The independent contribution of reciprocity is not sufficiently isolated, and the constructed preference labels are not empirically validated. Although the maximum scheduled training exposures are matched, the reported One-way DPO and ReMPA checkpoints are selected at 775 and 1,939 optimizer steps, respectively. Independent validation-based selection is reasonable for comparing practical performance, but this comparison does not isolate reciprocity under matched training exposure.
10 comments
AuthorsOpenReview ↗

Response to Reviewer aKxr(1/5): W1

We thank the reviewer for recognizing the practicality and theoretical motivation of ReMPA, and for the constructive suggestions. Below, we address key points including reciprocal loss weighting, continual learning comparisons, and the validity and controlled evaluation of reciprocal preferences.

W1. Sensitivity to Reciprocal Loss Weighting

The two directions in a reciprocal pair are symmetric by construction: each response is chosen under its corresponding molecular condition and rejected under the other. Neither direction has an inherent priority, so we assign them equal weights rather than introducing an arbitrary preference for one direction.

More importantly, equal weighting has a theoretical justification based on Appendix A. Under molecule-independent response reweighting, the two reciprocal directions have opposite β\beta-scaled margins, zz and −z-z. The weighted loss becomes

Lα(z)=−αlog⁡σ(z)−(1−α)log⁡σ(−z).L_\alpha(z)=-\alpha\log\sigma(z)-(1-\alpha)\log\sigma(-z).

For 0<α<10<\alpha<1, its optimum satisfies

\min_z L_\alpha(z)=H(\alpha).$$ When $\alpha=0.5$, the optimum is $z^\star=0$ with minimum loss $\log 2$, meaning that molecule-independent response reweighting alone cannot improve upon the reference-policy baseline. In contrast, when $\alpha\neq0.5$, we have $z_\alpha^\star\neq0$ and $H(\alpha)<\log 2$, allowing response-only bias toward the higher-weighted direction to reduce the loss without improving molecule–response compatibility. Although reducing the loss below $H(\alpha)$ still requires positive molecule–response interaction gain, unequal weighting no longer guarantees that any improvement over the reference-policy baseline requires such gain. Thus, equal weighting preserves the stronger baseline-relative guarantee established in Appendix A and makes the objective invariant to the ordering of molecules within a pair. While unequal weighting might affect the empirical adaptation–preservation trade-off, it introduces an additional tuning choice and permits loss reductions driven purely by molecule-independent response bias.
AuthorsOpenReview ↗

Response to Reviewer aKxr(2/5): W2

W2. Comparison with Continual Learning Methods

Regarding O-LoRA, its historical orthogonality constraint requires LoRA subspaces learned from previous tasks. Our setting starts directly from an off-the-shelf Llama-3.1-8B-Instruct model, without access to historical task-specific adapters. Therefore, the historical orthogonality term in the original method has no directly available target in our setting. Constructing such a history would introduce an additional training stage and associated data and computational costs.

We therefore added an SFT with general-instruction replay baseline using UltraChat-200k. UltraChat has been used for general-instruction replay with the same Llama-3.1-8B-Instruct backbone [1], evaluated as a replay baseline for medical-domain SFT [2], and included in the general-instruction corpus used by ChemLLM for chemical-domain adaptation [3]. We retain the original 50K molecular training examples and add 12.5K deduplicated, length-filtered first-turn instruction–response pairs from UltraChat-200k's train_sft split, making general instructions 20% of the mixed dataset.

All other training settings follow SFT-50K, with checkpoint selection based on the same molecular development set. Molecular-example exposure is preserved while additional general-instruction data are incorporated through replay.

ModelMoleculeQA Acc.↑FGBench Bool Avg.↑MMLU Micro Acc.↑Property Initial BA↑Property Initial Parse↑Property Repair BA↑
Base36.2452.0768.3648.6995.2151.63
SFT-50K41.3242.1366.914.769.5650.16
SFT-50K + Replay (20%)39.8920.7766.130.260.600.64
ReMPA41.2351.1068.2954.0994.8756.27

All metrics are percentages. The Base, SFT-50K, and ReMPA results are from Tables 5–6. MoleculeQA reports mean initial accuracy across three 10-shot support-set seeds; FGBench reports mean Boolean accuracy across its three reasoning categories; Property reports five-task macro-averaged balanced accuracy and initial valid-output rate.

In this configuration, proxy replay does not improve capability preservation. Output inspection shows that the severe FGBench and property-prediction degradation is associated with frequent task abstention and failure to produce the requested predictions. Although molecular SMILES are present in the property prompts, 889/898 responses begin with insufficient-information statements such as “I don't have the given molecule's SMILES code.” Only 7/898 initial outputs are parseable, increasing to 11/898 after repair. FGBench exhibits a similar pattern: Single and Interaction Boolean valid-output rates fall from 98.10% and 95.72% for SFT-50K to 45.33% and 37.68%, respectively.

A plausible contributing factor is response-style transfer from the replay corpus: 437/12,500 added instruction responses (3.50% of replay data and 0.70% of the full 62,500-example training set) begin with analogous insufficient-information statements, compared with none of the original 50K molecular responses under the same matching rule. This is consistent with inappropriate generalization of that response style to molecular tasks. The substantial domain and task-format gap between general-purpose and molecular instruction data may further limit the effectiveness of general-instruction replay for molecular adaptation.

These results support a more favorable adaptation–preservation trade-off for ReMPA than the evaluated general-instruction replay baseline, particularly in molecular reasoning and property prediction. ReMPA achieves this without requiring auxiliary general-instruction data or historical task adapters.

References

[1] Kotha S, Liang P. Replaying pre-training data improves fine-tuning. arXiv preprint arXiv:2603.04964. 2026 Mar 5.

[2] Ding F, Wang B. Improved supervised fine-tuning for large language models to mitigate catastrophic forgetting. arXiv preprint arXiv:2506.09428. 2025 Jun 11.

[3] Zhang D, Liu W, Tan Q, Chen J, Yan H, Yan Y, Li J, Huang W, Yue X, Ouyang W, Zhou D. Chemllm: A chemical large language model. arXiv preprint arXiv:2402.06852. 2024 Feb 10.

AuthorsOpenReview ↗

Response to Reviewer aKxr(3/5): W3

W3. Generalization Across Model Families

We initially used Llama-3.1-8B-Instruct as a shared backbone to isolate the effects of different adaptation strategies. To evaluate cross-family generalization, we additionally trained ReMPA and SFT-50K on Qwen3-8B-Base, alongside the unadapted Qwen model, using the non-thinking setting (enable_thinking=False).

The following table presents the cross-benchmark results (%) on Qwen3-8B-Base. MoleculeQA reports mean initial accuracy across three 10-shot support-set seeds; FGBench reports mean Boolean accuracy across its three reasoning categories; Property reports five-task macro-averaged balanced accuracy and initial valid-output rate.

ModelMoleculeQA Acc.↑FGBench Bool Avg.↑MMLU Micro Acc.↑Property Initial BA↑Property Initial Parse↑Property Repair BA↑
Base36.5146.8176.8732.9863.8143.93
SFT-50K42.3540.5876.592.413.564.84
ReMPA43.9251.8377.0134.2367.9645.75

The Qwen results reproduce the adaptation–preservation pattern observed on Llama. ReMPA improves MoleculeQA accuracy by 7.41 points over Base, compared with 5.84 points for SFT-50K. On FGBench, ReMPA improves over Base by 5.02 points, whereas SFT-50K declines by 6.23 points. ReMPA also maintains Base-level MMLU performance (77.01% vs. 76.87%).

For property prediction, SFT-50K exhibits low strict-format parse rates, primarily because it continues generating text after an answer label. To assess classification independently of output formatting, we additionally evaluated both adapted models on all 898 test examples using conditional log-likelihood scoring of the two candidate labels after a fixed Final answer: prefix, without free-form generation or format repair. Under this protocol, ReMPA achieves 52.98% macro BA, compared with 51.15% for SFT-50K. This indicates that ReMPA maintains an advantage in property prediction even when output-format parsing is eliminated. The substantial difference in valid-output rates additionally reflects ReMPA's stronger adherence to the required response format.

These results provide additional cross-family evidence that ReMPA can improve molecular capabilities while better preserving existing reasoning, general-purpose performance, and instruction-following behavior than conventional SFT.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.