- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ReMPA, a reciprocal preference optimization framework for molecular LLM adaptation. It constructs preference pairs from structurally similar molecules sharing the same instruction template, enabling domain adaptation without human preference annotations. Experiments on molecular comprehension, reasoning, property prediction, and general-purpose benchmarks suggest that ReMPA improves molecular capabilities while better preserving the base model's existing capabilities than conventional supervised fine-tuning (SFT). The problem is meaningful and the results are promising. However, concerns remain regarding the independent contribution of reciprocal preference construction, the reliability of pseudo-negative responses, and the fairness and robustness of the experimental comparisons.
Critical strengths
- Meaningful motivation. The paper addresses the important trade-off between acquiring domain-specific molecular capabilities and preserving the existing knowledge and reasoning abilities of general-purpose LLMs.
- Practical preference construction. ReMPA converts existing molecule–response supervision into reciprocal preference pairs without requiring additional human annotations or response generation.
- Comprehensive evaluation. The experiments cover multiple molecular and general-purpose benchmarks, with encouraging results demonstrating improved molecular performance and reduced capability degradation relative to SFT.
Critical weaknesses
W1. The contribution of reciprocal preference construction is insufficiently isolated. Although ReMPA outperforms One-way DPO on MoleculeQA and FGBench, their results on MMLU and molecular property prediction are nearly identical (Table 6). This raises the question of whether the primary benefit comes from reciprocal preference construction or preference optimization itself. More controlled comparisons with alternative preference construction strategies, matched training steps, and consistent checkpoint selection are needed.
W2. The reliability of pseudo-negative molecular responses is not sufficiently validated. ReMPA treats responses from structurally similar molecules as rejected alternatives. However, similar molecules may share properties, functional groups, and valid textual descriptions, making some pseudo-negative responses partially or entirely correct. The authors should quantify preference-label reliability and investigate how molecular similarity affects the quality of constructed preference pairs.
W3. The theoretical analysis does not directly support capability preservation. Appendix A shows that reciprocal preference optimization discourages molecule-independent response reweighting. However, this does not establish that the method preserves general-purpose or previously acquired molecular capabilities. Additional analysis of parameter drift, policy divergence, or prediction changes relative to the base model would strengthen the theoretical motivation.
W4. The SFT comparisons raise concerns about evaluation fairness. In Table 5, SFT-Full achieves an initial parse rate of only 5.96%, compared with 94.87% for ReMPA. Performance improves substantially after format repair, suggesting that output-format failures contribute significantly to the apparent degradation. Although the authors include a learning-rate-matched SFT control, stronger format-consistent and compute-matched comparisons are needed to isolate the benefits of ReMPA.
W5. The adaptation-preservation trade-off and statistical robustness require further evaluation. The main conclusions rely on selected training configurations, while differences between ReMPA and One-way DPO are relatively small on several benchmarks. Evaluating acquisition–preservation trade-off curves across training checkpoints and reporting variability across independent training runs would provide stronger evidence for the claimed advantages.
Questions for authors
Questions for the Authors
- What benefits does reciprocal preference construction provide beyond standard DPO under matched training conditions?
- Can the authors establish a clearer connection between the theoretical analysis and capability preservation?
- To what extent is the performance degradation of SFT caused by output-format failures rather than actual capability loss?
- Are the reported improvements consistent across multiple training seeds and different adaptation strengths?
8 comments
Response to Reviewer imWU(2/6): W2
W2. Reliability of Constructed Molecular Preference Labels
We conducted a three-stage empirical audit combining chemical entity verification, LLM-based semantic evaluation, and independent expert assessment to quantify preference-label reliability and examine its relationship with molecular similarity (Appendix B.3, Fig. 3).
1. Chemical entity extraction and structural verification. GLiNER2 extracts chemical entities from the original responses. An LLM agent generates or reuses entity-to-SMARTS mappings, while RDKit checks their structural validity and compatibility with the source molecules. Together, the LLM agent and RDKit support the revision or filtering of non-structural, contextually irrelevant, or problematic entities. The resulting entity–SMARTS table and source-matched substructure annotations provide auxiliary evidence for semantic evaluation.
2. LLM-based semantic preference audit. We sampled 500 complete reciprocal pairs (1,000 directional preferences), balanced across instruction templates, and evaluated both directions separately. For each direction, the LLM judge receives the instruction, target SMILES, two anonymized responses, and auxiliary entity/SMARTS annotations. It compares structural accuracy, chemical and biological explanations, and instruction relevance, selecting the better response or AMBIGUOUS when no clear quality difference exists. Original preference labels are hidden, and candidate positions are balanced.
We further analyzed Morgan-fingerprint Dice similarity across judgment categories:
| LLM semantic judgment | Examples | Proportion (%) | Mean Dice ± SD | Median Dice [P25, P75] |
|---|---|---|---|---|
| Chosen preferred | 831 | 83.10 | 0.7264 ± 0.1247 | 0.7229 [0.6197, 0.8280] |
| Rejected preferred | 118 | 11.80 | 0.7999 ± 0.1206 | 0.8373 [0.7098, 0.8965] |
| Ambiguous | 51 | 5.10 | 0.8330 ± 0.1079 | 0.8834 [0.7864, 0.9111] |
| Total | 1,000 | 100.00 | — | — |
The LLM judge supports 83.1% of the original preference labels, favors the rejected response in 11.8%, and identifies 5.1% as ambiguous.
Structural similarity is associated with preference-label reliability. Mean Dice similarity increases from 0.7264 for chosen-preferred cases to 0.7999 for rejected-preferred cases and 0.8330 for ambiguous cases. This trend is consistent with greater overlap in functional groups, molecular properties, and valid descriptions among highly similar molecules. These findings highlight a trade-off in similarity-based preference construction: structurally similar molecules provide plausible and challenging pseudo-negatives, while greater structural overlap can make molecule-conditioned preferences more difficult to distinguish.
3. Independent expert validation. We further selected 55 directional examples from distinct reciprocal pairs, approximately preserving the LLM judgment distribution. An independent expert evaluated anonymized responses using the instruction, target molecule's IUPAC name, and RDKit-generated 2D structure, without access to preference labels or LLM judgments.
The expert favored the originally chosen response in 51/55 cases (92.73%), the rejected response in 1/55 (1.82%), and assigned AMBIGUOUS in 3/55 (5.45%).
| Expert \ LLM | Chosen | Rejected | Ambiguous | Total |
|---|---|---|---|---|
| Chosen | 45 | 5 | 1 | 51 |
| Rejected | 0 | 1 | 0 | 1 |
| Ambiguous | 1 | 0 | 2 | 3 |
| Total | 46 | 6 | 3 | 55 |
Expert and LLM judgments agree on 48/55 cases (87.27%). Five of the seven disagreements involve the LLM favoring the rejected response while the expert favors the original chosen response. This suggests that the LLM-based audit may overestimate the frequency of preference reversals relative to independent expert judgments in this subset. The expert audit also shows higher mean Dice similarity for ambiguous cases (0.8970) than chosen-preferred cases (0.7118), consistent with the LLM-based trend.
Overall, the audits support the constructed preferences in 83.1% of LLM judgments and 92.73% of expert judgments, with 87.27% expert–LLM agreement. The similarity analysis further identifies high structural similarity as a factor associated with ambiguous and reversed preferences, providing empirical insight into the trade-off between challenging pseudo-negatives and preference-label reliability.
Response to Reviewer imWU(3/6): W3 & Q2
W3 & Q2. Connection Between Theoretical Analysis and Capability Preservation
Appendix A provides a theoretical explanation of how reciprocal preference construction shapes the adaptation process. Proposition 1 establishes that molecule-independent response reweighting cannot reduce the symmetric reciprocal loss below the reference-policy baseline. Proposition 2 further shows that improving beyond this baseline requires positive molecule–response interaction gain , while Eq. (40) demonstrates that the initial optimization gradient is directly proportional to . Together, these results explain how reciprocal supervision encourages molecule-conditioned preference learning rather than globally favoring or suppressing particular responses.
Training dynamics provide additional evidence consistent with this mechanism. Our logged reference-relative likelihoods reveal a notable difference between One-way DPO and ReMPA. At their selected checkpoints, One-way DPO (step 775) exhibits a chosen log-probability change of −19.01 and a rejected change of −59.33, indicating that its preference margin improves even as the chosen-response likelihood decreases substantially. In contrast, ReMPA (step 1,939) exhibits a much smaller chosen decrease (−4.39) alongside a substantial rejected decrease (−70.69). This difference becomes more pronounced over training: at step 1,939, the One-way DPO chosen log-ratio reaches −107.37, compared with −4.39 for ReMPA, as measured on their respective validation preference sets. These observations suggest that ReMPA achieves preference separation with substantially less chosen-response likelihood displacement, rather than relying heavily on suppressing both preferred and rejected responses.
The additional Non-reciprocal DPO-50K control provides complementary evidence at the capability level. Despite comparable molecular comprehension performance (41.23% vs. 40.52% on MoleculeQA), ReMPA achieves 51.10% FGBench Boolean accuracy, compared with 46.37% for the non-reciprocal control. Relative to the Base model (52.07%), this corresponds to a degradation of only 0.97 percentage points for ReMPA, versus 5.70 points for the non-reciprocal control. This indicates substantially stronger preservation of previously acquired molecular reasoning performance under comparable adaptation gains.
Together, these findings provide a theoretical and empirical connection between reciprocal preference learning and capability preservation. The theoretical analysis shows that reciprocal supervision encourages molecule-conditioned preference discrimination rather than molecule-independent response reweighting. Consistent with this mechanism, the training logs indicate that ReMPA achieves preference separation with substantially smaller chosen-response likelihood displacement than One-way DPO, avoiding the pronounced suppression of preferred responses observed in the latter. This more conservative adaptation behavior is consistent with the substantially better retention of molecular reasoning performance observed in FGBench. These findings suggest that reciprocal preference learning can acquire molecule-specific distinctions while reducing unnecessary disruption to capabilities already present in the base model.
Response to Reviewer imWU(1/6): W1 & Q1
We thank the reviewer for recognizing the significance of our motivation, the practicality of our approach, and the comprehensive evaluation, and for the constructive suggestions. Below, we address key points including the contribution and reliability of reciprocal preferences, their connection to capability preservation, and evaluation fairness.
W1 & Q1. Controlled Comparison of Preference Construction Strategies
We thank the reviewer for highlighting the need to better isolate the contribution of reciprocal preference construction. To provide a more controlled comparison, we conducted an additional Non-reciprocal DPO-50K experiment that preserves the full positive-record coverage of ReMPA while removing reciprocal supervision.
Specifically, we preserved all 50,000 original training condition–response records, including their system prompts, molecular conditions, instruction templates, and chosen responses, and changed only the rejected responses. Each chosen record appears exactly once. Donors are selected from the original records within the same template and split, with a different canonical molecule. Self-pairing and reciprocal two-cycles are prohibited across the entire dataset. Thus, an assignment
a → bexcludes the reverse assignmentb → athroughout training.To maintain structurally plausible negatives, we maximized total Morgan-fingerprint Dice similarity under these constraints and the training-length requirements. The resulting non-reciprocal control achieves a mean training Dice similarity of 0.6585, compared with 0.7379 for ReMPA. Both variants use the same backbone, LoRA configuration, batch settings, and optimizer settings. Reference log-probabilities were recomputed for the new preferences. Our validation-based checkpoint-selection procedure selected the non-reciprocal checkpoint at step 1,955, close to the submitted ReMPA checkpoint at step 1,939.
The following table compares the new control with the submitted Table 6 results. All values are percentages. MoleculeQA accuracy is averaged over support-set seeds 0, 42, and 123, while Property BA is macro-averaged across five tasks.
The non-reciprocal control improves MoleculeQA accuracy over One-way DPO (40.52% vs. 39.83%). ReMPA further outperforms the coverage-matched non-reciprocal control by 0.71 percentage points on MoleculeQA, 4.73 points on FGBench, and 5.54/3.81 points on initial/repaired property BA, while MMLU remains comparable across all three models.
These results extend our original One-way DPO ablation with a stronger control for positive-record coverage and closely matched training updates. ReMPA achieves consistently stronger molecular performance than the similarity-optimized non-reciprocal control, while maintaining comparable general-capability performance. This comparison provides additional empirical evidence for the value of reciprocal preference construction in molecular adaptation.