- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper reframes reaction-condition recommendation as budget-constrained "variable-space design": given a reaction and an open library of candidate reagents organized into functional categories (catalyst, ligand, base, solvent), select per-category subsets whose Cartesian product is a compact screening subspace that retains high-yield combinations under an experimental budget. The proposed method, ChemVD, encodes reactions (DRFP), categories (frozen Llama-3.1-8B) and reagents (Morgan fingerprints) into a shared embedding space, retrieves categories first, then ranks reagents with a category-conditioned query jointly encoding reaction and category. Training uses multi-positive contrastive learning, a per-positive coverage loss, yield-aware positive weighting, and several chemically informed regularizers (low-yield suppression, explicit-negative margin loss, category reweighting, frequency-based hard negatives). Validation: (i) Reagent-Split retrieval on a ChemRANG-derived dataset, stratified into seen/unseen reagents; (ii) HTE-OOD subspace design on 15 Suzuki high-throughput subsets expanded with 24 infeasible salt "catalysts" (from 384 to 9,600 conditions), evaluated by mean achievable maximum yield (MAMY) under per-category budgets; (iii) a controlled XOR / Independent distribution experiment and same-encoder ablations isolating the hierarchical architecture. Headline: at a 20% budget, ChemVD (8B backbone) reaches MAMY 68.78%, beating the zero-shot DeepSeek-V4-Pro (1.6T) API baseline by 20.44 pp, while the same-encoder Pairwise ablation reaches only ~15% mean MAMY.
Critical strengths
- A genuinely useful reframing, validated by the right control experiment. Recasting condition recommendation as budget-constrained open-candidate retrieval for compact subspace construction seems to be novel (I could not find this formulation in the literature) and directly matches how experimentalists actually design screens. Crucially, the paper isolates the architectural contribution with a same-encoder Pairwise ablation (identical encoders, data, multi-positive objective; only the hierarchical query removed): mean MAMY 15.27% vs 78.14%, with consistent wAPC@K gains on both seen and unseen reagent retrieval, plus a clean XOR / Independent distribution experiment. This is the strongest evidence in the paper and it supports the central claim that joint reaction–category conditioning, not candidate-space openness alone (BCE also has open candidates and fails), drives the gains.
- Above-average experimental coverage. Baselines span closed-set classifiers (ReaCon, Label-mix), autoregressive models (Parrot, TextReact), zero-shot LLMs from 30B to 1.6T (local and API, with verbatim prompts), a nonparametric train-frequency baseline, and objective-level baselines; ablations cover every loss component, encoder swaps (Table 13), and data volume. Leakage-aware cluster splitting, shared candidate pools for the seen / unseen strata, three seeds with standard deviations, and full hyperparameter tables show real care.
Critical weaknesses
- The headline LLM comparison is misleading as presented. The abstract's "outperforming a 1.6T-parameter LLM baseline by 20.44 percentage points" selects the weakest LLM (DeepSeek-V4-Pro) at the single budget where it is weakest. Table 12 shows Llama-3.1-70B (71.26) and Llama-3.3-70B (72.16) both beat ChemVD (68.78) at the 20% budget, and Llama-3.3-70B also leads at 30%; ChemVD's wins at 40-80% budgets occur where the task is near-trivial (even Pairwise reaches 85.99 at 80%). The body text is honest about this, so the full data is available — but the abstract overstates the result. Relatedly, Figure 3(b) plots "model parameters" including ChemVD's frozen Llama-3.1-8B encoder, so "an 8B model beats 70B/1.6T LLMs" really means "a small trainable head on a frozen 8B encoder, trained on ~98k domain examples, beats zero-shot general-purpose LLMs" — a legitimate and interesting result that the figure's framing obscures. The OOD evidence is weaker than claimed. The HTE-OOD expansion adds 24 inorganic salts (acetates, chlorides, bromides, fluorides, sulfates, BF4-/PF6-) as pseudo-yield-0 catalysts — candidates no chemist would consider for Suzuki coupling. The demonstrated "generalization to unseen reagents" is therefore mostly the ability to exclude trivially infeasible salts, not ranking of plausible unseen reagents (e.g., other Pd complexes or new ligands). The abstract's claim that ChemVD "generalizes effectively to reagents unseen during training" rests mainly on Reagent-Split Unseen (n=1,100), where the margin over Pairwise narrows substantially. Additionally, MAMY only requires that one high-yield combination survive in the 20-condition subspace — a lenient metric with no complementary measure of overall subspace quality.
Questions for authors
- Table 12: Pairwise MAMY is 0.00±0.00 at 20 / 30 / 40% budget, then jumps to 1.60 at 50% and 85.99 at 80%. What causes this cliff? Does Pairwise consistently retain the same solvent / base (so only catalyst ranking matters at high budget)? Please explain or show per-category retention.
- The abstract claims ChemVD "generalizes effectively to reagents unseen during training." Can you evaluate subspace design on an OOD axis with plausible unseen catalysts (e.g., Pd complexes or ligands absent from training but mechanistically valid), rather than infeasible salts? Would the 20.44 pp gap over LLMs hold there?
- Is code and the processed ChemRANG split available (or will be, upon acceptance)? I could not find any availability statement in the paper.
- A.5.4: MAMY is highest at 10% training data, drops sharply at 20%, then fluctuates. What explains the non-monotonicity — is it seed variance, or does more data shift rankings toward historically frequent (but lower-yield) conditions?
- The category ground truth is constant per reaction family (every Suzuki reaction has {Catalyst, Ligand, Base, Solvent}). The "58.1% improvement" in All-truth hit@5 therefore measures ranking these four above distractors, not learning which roles a reaction requires. Do you agree this evaluation is near-trivial, and can you report category results on a setting where the required role set varies (e.g., reactions needing additives but no ligand)?
- Can you add some kind of a complementary subspace-quality metric? For instance, fraction of >90%-yield conditions retained, or mean yield over the selected subspace. That would make MAMY's leniency (one surviving high-yield combination suffices) explicit.
- All subspace-design evidence is Suzuki. Can the framework transfer to other reaction families (the training set includes Michael and indole) or to non-discrete variables such as temperature / time?
9 comments
Response to Reviewer 3 (2/8): Reagent Retention and the Abrupt Increase in Pairwise MAMY
Q1: Table 12: Pairwise MAMY is 0.00±0.00 at 20 / 30 / 40% budget, then jumps to 1.60 at 50% and 85.99 at 80%. What causes this cliff? Does Pairwise consistently retain the same solvent / base (so only catalyst ranking matters at high budget)? Please explain or show per-category retention.
A1: We thank the reviewer for this insightful question. To investigate the abrupt increase in MAMY, we inspected the category-wise reagent rankings and tracked how the retained candidate sets evolve across screening budgets. Since presenting full retention breakdowns for every HTE-OOD subset would be impractical, and their MAMY trajectories closely mirror the overall trend (Table R3.1), we reported Sub3-neg as a representative case.
Our analysis reveals that the abrupt increase is primarily associated with the delayed inclusion of a critical catalyst (Table R3.2). Specifically, the candidate pools contain 25 catalysts, 12 ligands, 8 bases, and 4 solvents. Notably, Pairwise ranks Pd(OAc)₂, the sole effective catalyst in Sub3-neg, 17th among 25 catalyst candidates. Consequently, Pd(OAc)₂ is excluded at screening budgets of 60% or lower and is only retained when the budget reaches 80%. By contrast, key ligands such as PPh₃ and P(o-tol)₃ are already retained at much smaller budgets. Furthermore, due to the relatively small candidate pools for bases and solvents, increasing the screening budget introduces only a limited number of additional candidates in these categories. Together, these findings explain the abrupt MAMY increase and highlight how category-wise ranking errors can bottleneck reaction discovery under limited screening budgets.
Table R3.1: Pairwise MAMY (mean ± standard deviation) across HTE-OOD subsets at different screening budgets.
| Budget | MAMY |
|---|---|
| 20%–40% | 0.00 ± 0.00 |
| 50% | 1.60 ± 6.20 |
| 60% | 4.03 ± 8.90 |
| 80% | 85.99 ± 15.32 |
Table R3.2:Pairwise reagent rankings by category and retention cutoffs in Sub3-neg.
| Rank | Catalyst (N=25) | Ligand (N=12) | Base (N=8) | Solvent (N=4) |
|---|---|---|---|---|
| 1 | Sr(OAc)₂ | PPh₃ | Et₃N | THF (20%) |
| 2 | Mg(OAc)₂ | AmPhos (20%) | LiOtBu (20%) | DMF (40/50%) |
| 3 | NaOAc | PCy₃ | NaHCO₃ (40%) | MeCN (80%) |
| 4 | KOAc | dppf | K₃PO₄ (50%) | MeOH |
| 5 | Ca(OAc)₂ (20%) | P(o-tol)₃ (40%) | None | -- |
| 6 | LiOAc | Xantphos (50%) | CsF (80%) | -- |
| 7 | Ba(OAc)₂ | dtbpf | NaOH | -- |
| 8 | Al(OAc)₃ | cataCXium A | KOH | -- |
| 9 | NaBF₄ | None | -- | -- |
| 10 | NaPF₆ (40%) | P(t-Bu)₃ (80%) | -- | -- |
| 11 | KBF₄ | SPhos | -- | -- |
| 12 | Na₂SO₄ | XPhos | -- | -- |
| 13 | NaF (50%) | -- | -- | -- |
| 14 | KPF₆ | -- | -- | -- |
| 15 | KF | -- | -- | -- |
| 16 | K₂SO₄ | -- | -- | -- |
| 17 | Pd(OAc)₂ | -- | -- | -- |
| 18 | LiBr | -- | -- | -- |
| 19 | NaBr | -- | -- | -- |
| 20 | KBr (80%) | -- | -- | -- |
| 21 | LiCl | -- | -- | -- |
| 22 | NaCl | -- | -- | -- |
| 23 | KCl | -- | -- | -- |
| 24 | MgCl₂ | -- | -- | -- |
| 25 | CaCl₂ | -- | -- | -- |
Response to Reviewer 3 (3/8): Generalization to Chemically Plausible Unseen Ligands
W1 & Q2: The abstract claims ChemVD "generalizes effectively to reagents unseen during training." Can you evaluate subspace design on an OOD axis with plausible unseen catalysts (e.g., Pd complexes or ligands absent from training but mechanistically valid), rather than infeasible salts? Would the 20.44 pp gap over LLMs hold there?
A2: We thank the reviewer for this important suggestion. Additional evaluation was conducted using unpublished experimental data from a 1,4-diene asymmetric 1,3-difunctionalization reaction newly developed by our collaborators. As this work is unpublished and involves confidential information, we cannot currently disclose the specific reactions, complete reaction-condition records, or reagent structures. We therefore report only the training setup and selected key evaluation results below.
In this reaction, a total of 12 chiral ligands were experimentally tested, among which TY-Phos series ligands, especially TY2, TY3, TY4, and TY5, performed well in the wet-lab experiments, with TY5 as the best-performing ligand. Other TY-phos ligands (TY1, TY6-9) exhibited lower performance, and the remaining Ming-phos (M1), Xu-phos (Xu1), and Xiang-phos (X1) ligands could barely promote the reaction. However, all 12 ligands are plausible choices for chemists prior to experiments, which provides us with a challenging scenario to evaluate ChemVD.
For model training, we collected 177 experimental records from this new reaction system, all involving TY5 as the ligand, and jointly trained ChemVD with the original Suzuki dataset using the same model configuration as in the manuscript. For evaluation, we considered seven reactions, including three represented in the training data and four previously unseen substrate pairs. The inference candidate pool contained 79 reagents, of which 68 were present in the training data, and 11 were newly introduced ligands entirely unseen during training. Specifically, among the 12 candidate ligands, only TY5 appeared in the training data, whereas the remaining 11 ligands, including the experimentally effective TY2, TY3, and TY4, were unseen. This setting allows us to assess whether ChemVD could choose privileged ligands (TY2-TY5) over other ligands. ChemVD and the LLM baselines were evaluated using identical candidate pools and evaluation conditions.
We evaluated recommendation performance using Exact@1 and Family@K. Exact@1 measures whether the top-ranked recommendation matches the experimentally identified reference combination, while Family@K measures the proportion of the top-K recommendations belonging to the experimentally validated ligand family (TY2-TY5). As shown in Table R3.3, ChemVD achieves an Exact@1 of 0.77 and Family@1, Family@3, and Family@4 scores of 1.00, 0.81, and 0.72, respectively, consistently outperforming all evaluated LLM baselines.
These results provide additional evidence that ChemVD can distinguish chemically plausible reagents with different experimental performance, including promising ligands entirely unseen during training, rather than merely excluding obviously infeasible candidates.
Table R3.3: Reagent recommendation performance for 1,4-diene asymmetric 1,3-difunctionalization.
| Model | Exact@1 | Family@1 | Family@3 | Family@4 |
|---|---|---|---|---|
| Qwen3.5-35B | 0.08 | 0.37 | 0.47 | 0.48 |
| Qwen3-30B | 0.00 | 0.42 | 0.48 | 0.49 |
| Llama3.1-70B | 0.08 | 0.48 | 0.52 | 0.52 |
| Llama3.3-70B | 0.03 | 0.52 | 0.48 | 0.47 |
| DeepSeek-v4-pro | 0.21 | 0.56 | 0.67 | 0.65 |
| ChemVD | 0.77 | 1.00 | 0.81 | 0.72 |
Response to Reviewer 3 (1/8): Description of Parameter Efficiency Advantages and LLM Comparisons
We thank the reviewer for the positive assessment of our work and for recognizing its potential value. We also appreciate the constructive suggestions. In response, we have refined our claims regarding LLM comparisons and conducted additional experiments and analyses on generalization to chemically plausible unseen reagents, category prediction under different reagent-role requirements, and subspace design quality. We provide detailed responses to each concern below.
W1: The headline LLM comparison is misleading as presented. The abstract's "outperforming a 1.6T-parameter LLM baseline by 20.44 percentage points" selects the weakest LLM (DeepSeek-V4-Pro) at the single budget where it is weakest. Table 12 shows Llama-3.1-70B (71.26) and Llama-3.3-70B (72.16) both beat ChemVD (68.78) at the 20% budget, and Llama-3.3-70B also leads at 30%; ChemVD's wins at 40-80% budgets occur where the task is near-trivial (even Pairwise reaches 85.99 at 80%). The body text is honest about this, so the full data is available — but the abstract overstates the result.
A W1: We thank the reviewer for this suggestion. We have revised the abstract to highlight that ChemVD achieves competitive subspace design performance through domain-specific representation learning with a substantially smaller frozen backbone, rather than claiming superiority over all zero-shot LLM baselines, as follows:
“On HTE-OOD, under a 20% screening budget, ChemVD achieves a mean achievable maximum yield of 68.78% through domain-specific representation learning with a frozen 8B-parameter backbone, closely competitive with the strongest LLM baseline (trailing by 3.38 percentage points) while outperforming the 1.6T-parameter LLM baseline by 20.44 percentage points.”
...Relatedly, Figure 3(b) plots "model parameters" including ChemVD's frozen Llama-3.1-8B encoder, so "an 8B model beats 70B/1.6T LLMs" really means "a small trainable head on a frozen 8B encoder, trained on ~98k domain examples, beats zero-shot general-purpose LLMs" — a legitimate and interesting result that the figure's framing obscures.
To avoid potential confusion regarding model size and training settings, we have revised the caption of Figure 3(b) to explicitly distinguish ChemVD’s domain-specific representation learning from the zero-shot evaluation of general-purpose LLM baselines, as follows:
“Figure 3(b) Average MAMY versus backbone model size. ChemVD employs a frozen 8B-parameter backbone with a domain-trained representation learning module, whereas the general-purpose LLM baselines are evaluated in a zero-shot setting.”