- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies structure elucidation from experimental ¹H/¹³C NMR chemical-shift lists. It fine-tunes Qwen3.5-4B jointly on spectrum→SMILES and SMILES→spectrum (NMR-LM), then applies DPO with ground-truth positives and the most similar formula-matched incorrect sample as the negative. On top of NMR-LM, NMR-CRAFT is a test-time search: candidates from beam search (32), sampling (32) and spectral retrieval from SimNMR-PubChem (50, using NMR-Solver's database and retrieval) are fragmented; fragment pairs whose heavy-atom counts sum to the target formula are reassembled; up to 100 candidates are preselected by a rank fusion of parent-source priors and recombination support; and the final ranking mixes this prior with the mean log-likelihood of the observed spectrum under NMR-LM's forward direction. On scaffold-split NMRGym with Exclude-Scaffold retrieval, Top-1 rises from 3.75% (NMR-LM, beam 10) to 8.23%; on random-split NMRSpec, from 47.80% to 58.51%. On a 1,000-molecule NMRGym subset, NMR-CRAFT reaches 8.1% Top-1 in 14.2 s per molecule versus 2.9% and 222–260 s for a reproduced NMR-Solver under database exclusion. NMR-LM is also evaluated for spectrum prediction with set- and vector-similarity metrics.
Critical strengths
-
Careful leakage control and unusually complete reproducibility. The paper recognises that SimNMR-PubChem contains 74.9% of NMRGym and 33.2% of NMRSpec test molecules (l. 340), applies Exclude-Scaffold / Exclude-Molecule masks during retrieval (Appendix C.1), and reports how sharply NMR-Solver drops when test molecules are removed (Table 5: 12.50% → 2.90% Top-1). The authors release the checkpoint, the full NMR-CRAFT outputs and the code. This makes the numbers checkable and is a real service to a field where leakage through large spectral databases is common.
-
The forward-likelihood verification signal is shown to matter. Removing spectral likelihood from the final ranking halves Top-1 on the 1,000-molecule subset (8.1% → 4.1%, Table 22), and both endpoints of the λ_rank sweep (4.1% and 4.3%) are far below the mixed setting (Fig. 9f). Together with the NMRSpec stratification by molecular size and training-set similarity with bootstrap CIs (Fig. 3a/b, Table 14), this gives credible evidence that a bidirectionally trained model provides a useful candidate-verification signal, including for larger and less familiar molecules.
Critical weaknesses
1. The comparison with NMR-Solver is not "under the same protocol" (as claimed in l. 087, 099 and 483), so the accuracy and 15× speed claims are not attributable to the method.
- Formula access differs. NMR-CRAFT uses the molecular formula as a hard constraint inside the search: fragment pairs must sum to the target heavy-atom counts and products must match the formula exactly (Eq. 7; l. 218–228). For NMR-Solver, only the allowed element types are given, and the formula is applied only as an output filter that does not modify the internal FB-MO search (Appendix B.6, l. 902–911). In structure elucidation the formula is a very strong constraint, so this asymmetry alone could explain much of the gap.
- A constraint of NMR-Solver is switched off. Hydrogen-multiplicity constraints are disabled (l. 903).
- Budgets and hardware differ. NMR-Solver is run with 1,000 retrieved candidates, up to 200,000 fragment pairs and a 1,000-molecule pool (Table 10), whereas NMR-CRAFT uses 50 retrieved candidates, 1,000 pairs and 100 candidates. NMR-CRAFT's two model services run on separate A100s of a 4-GPU server (l. 829–831), while the NMRNet ensemble runs on a single A100 (l. 899–900). The 15× speed-up therefore largely reflects budget choices rather than an algorithmic advantage; an accuracy–time curve at matched budgets is needed.
- The reproduction looks weak. Unrestricted full NMR-Solver obtains 12.50% Top-1 (Table 5), lower than its own search-only component with formula filtering in Table 1 (17.92%). The subsets differ, but a full pipeline performing much worse than its retrieval stage suggests a configuration problem that needs to be explained.
2. Search hyper-parameters appear to have been tuned on the same test molecules that are used to report results, and most workflow ablations are within noise.
- Appendix D.2 (l. 1263–1273) states that the five search budgets and λ_rank were varied on the same 1,000 NMRGym test molecules used for Table 5. The defaults (λ_rank = 0.32, source weights 1.00/0.65/0.35, δ = 4, κ = 60; l. 841, 871, 884) are not said to be chosen on validation data, and the chosen λ_rank is exactly the best value in that test-set sweep. If so, Tables 1, 5 and 22 are optimistically biased.
- On 1,000 molecules at p ≈ 0.08, the standard error of Top-1 is about 0.86 points. The ablations "w/o database" (−0.9), "w/o sampling" (−0.7) and "w/o support" (−1.0) in Table 22 are each about one standard error (7–10 molecules), and no paired test is reported. Only the likelihood ablation is clearly significant, so the claim that database retrieval, sampling and the support signal are complementary (l. 466–468) is not established.
- The headline "120% improvement over direct generation" compares against beam-10 decoding. Table 4 shows that 32-beam candidates with formula filtering and likelihood re-ranking already give 5.36%, so retrieval plus recombination adds about 2.9 points (≈54%). With 8.23% Top-1 and 17.34% Top-10 on scaffold-split NMRGym, the description as a potentially practical tool (l. 032) overstates the current capability.
Questions for authors
Clarifications that could change my assessment
- Were λ_rank, the source weights, δ, κ, B_pair and B_cand chosen on the NMRGym validation split or on test molecules? Please report test results with all search hyper-parameters selected on validation data only.
- Can NMR-Solver's official implementation take the molecular formula as a search constraint (and hydrogen multiplicities)? If so, please re-run Table 5 with these enabled and with candidate/pair budgets matched to NMR-CRAFT on the same hardware, and report accuracy versus wall-clock time.
- Why does full NMR-Solver without exclusion (12.50% Top-1, Table 5) underperform its own search-only + formula component (17.92%, Table 1)?
- Please report paired confidence intervals (e.g., bootstrap or McNemar) for the Table 22 ablations.
- Spectrum prediction: for per-atom predictors (CASCADE, DetaNet, NMRNet, Mnova), were symmetry-equivalent atoms merged to match the reference format of one ¹³C shift per distinct environment? How many molecules failed for each method, and what are the results on the subset where all methods produce a prediction? Please also report per-atom MAE (ppm), which is the standard metric in this area.
- On NMRSpec, do the NMRTrans/NMRMind numbers (taken from the original papers) use the molecular formula and the multiplicity/J-coupling information that NMR-LM receives (Appendix B.1)? Since you reproduced NMRTrans (Appendix D.1), please also report it on NMRGym.
Suggestions (not required for my assessment)
- How are acyclic molecules (empty Bemis–Murcko scaffold) handled by Exclude-Scaffold, and is Exclude-Molecule based on stereo-aware SMILES or InChIKey connectivity? Does exact match require correct stereochemistry?
- Please qualify "state-of-the-art" in the abstract as in the introduction ("among the evaluated generative models"), and state the absolute Top-1/Top-10 next to the relative gain.
8 comments
Response to Reviewer sAmq (Part 2/3)
W1-4: The reproduction looks weak. Unrestricted full NMR-Solver obtains 12.50% Top-1 (Table 5), lower than its own search-only component with formula filtering in Table 1 (17.92%). The subsets differ, but a full pipeline performing much worse than its retrieval stage suggests a configuration problem that needs to be explained.
Q3: Why does full NMR-Solver without exclusion (12.50% Top-1, Table 5) underperform its own search-only + formula component (17.92%, Table 1)?
Thank you for raising this concern. Table 1 reports results on the full NMRGym test set (27,054 molecules), whereas Table 5 uses a 1,000-molecule subset. For a direct comparison, we evaluated the formula-filtered search-only baseline on the same subset.
| Setting | Top-1 | Top-5 | Top-10 |
|---|---|---|---|
| Search-only + Formula | 13.9 | 32.7 | 35.7 |
| Full NMR-Solver | 12.5 | 31.5 | 36.5 |
The Top-1 difference is 1.4 percentage points. Among the 31 cases correctly ranked first by search-only but not by the full pipeline, all ground-truth structures remain in the final candidate lists but are ranked lower. Conversely, the full pipeline correctly ranks 17 additional molecules first and achieves slightly higher Top-10 accuracy. Thus, the difference primarily reflects changes in candidate ranking rather than the loss of correctly retrieved structures.
W2-1: Appendix D.2 (l. 1263–1273) states that the five search budgets and λ_rank were varied on the same 1,000 NMRGym test molecules used for Table 5. The defaults (λ_rank = 0.32, source weights 1.00/0.65/0.35, δ = 4, κ = 60; l. 841, 871, 884) are not said to be chosen on validation data, and the chosen λ_rank is exactly the best value in that test-set sweep. If so, Tables 1, 5 and 22 are optimistically biased.
Q1: Were λ_rank, the source weights, δ, κ, B_pair and B_cand chosen on the NMRGym validation split or on test molecules? Please report test results with all search hyper-parameters selected on validation data only.
Thank you for raising this concern. The search hyperparameters were selected using validation data, not test data. Specifically, we selected Beam/Sampling/Database budgets (32/32/50), B_pair (1,000), B_cand (100), and λ_rank (0.32) based on Top-10 sequence accuracy on a subset of 2,000 NMRGym validation molecules. The source weights (1.00/0.65/0.35) and smoothing constants (δ = 4, κ = 60) were fixed heuristically rather than separately tuned.
The test-set sensitivity analysis in Appendix Fig. 9 was conducted only to assess robustness after parameter selection. We have replaced it with the validation-set results and clarified that all reported test results use hyperparameters fixed independently of test-set performance.
W2-2: On 1,000 molecules at p ≈ 0.08, the standard error of Top-1 is about 0.86 points. The ablations "w/o database" (−0.9), "w/o sampling" (−0.7) and "w/o support" (−1.0) in Table 22 are each about one standard error (7–10 molecules), and no paired test is reported. Only the likelihood ablation is clearly significant, so the claim that database retrieval, sampling and the support signal are complementary (l. 466–468) is not established.
Q4: Please report paired confidence intervals (e.g., bootstrap or McNemar) for the Table 22 ablations.
Thank you for raising this concern. We computed 95% paired bootstrap confidence intervals using 10,000 resamples of the same 1,000 NMRGym test molecules. The sequence accuracy differences (ablation minus full NMR-CRAFT, in percentage points) are reported below.
| Ablation | Top-1 Δ [95% CI] | Top-10 Δ [95% CI] |
|---|---|---|
| w/o Database | −0.9 [−2.0, 0.2] | −2.1 [−3.4, −0.8] |
| w/o Sampling | −0.7 [−1.5, 0.1] | −1.2 [−2.3, −0.1] |
| w/o Support | −1.0 [−1.8, −0.2] | −0.8 [−1.7, 0.1] |
| w/o Likelihood | −4.0 [−5.6, −2.5] | −3.5 [−4.8, −2.3] |
Removing spectral likelihood produces the largest and most consistent degradation. The other components show smaller, metric-dependent effects, with some confidence intervals including zero. Together with the fragment-source analysis in Fig. 5(a,b), these results provide further evidence for the complementary roles of candidate sources and ranking signals. We have added the complete paired confidence intervals to Appendix D.3.
Response to Reviewer sAmq (Part 3/3)
W2-3: The headline "120% improvement over direct generation" compares against beam-10 decoding. Table 4 shows that 32-beam candidates with formula filtering and likelihood re-ranking already give 5.36%, so retrieval plus recombination adds about 2.9 points (≈54%). With 8.23% Top-1 and 17.34% Top-10 on scaffold-split NMRGym, the description as a potentially practical tool (l. 032) overstates the current capability.
Thank you for raising this concern. As the reviewer noted, the reported 120% improvement is calculated relative to the NMR-LM Beam-10 baseline in Table 1. We agree that the stronger Beam-32 + Formula + Likelihood baseline provides additional context, against which NMR-CRAFT improves Top-1 accuracy from 5.36% to 8.23% (+2.87 percentage points).
While absolute accuracy on scaffold-split NMRGym remains limited, NMR-CRAFT improves Top-10 accuracy from 9.51% to 17.34% on NMRGym and from 66.48% to 79.67% on NMRSpec, demonstrating its value for candidate refinement. We have clarified the comparison in the abstract and added the Beam-32 + Formula-filter baseline to Table 1.
Q5: Spectrum prediction: for per-atom predictors (CASCADE, DetaNet, NMRNet, Mnova), were symmetry-equivalent atoms merged to match the reference format of one ¹³C shift per distinct environment? How many molecules failed for each method, and what are the results on the subset where all methods produce a prediction? Please also report per-atom MAE (ppm), which is the standard metric in this area.
Regarding symmetry-equivalent atoms, we adopt the aggregate results reported by NMRGym. Its paper describes Hungarian matching for peak-set evaluation but does not explicitly specify whether symmetry-equivalent atoms are merged before matching. For the MAE comparison, we use Hungarian matching with an L1 chemical-shift distance.
According to the coverage reported by NMRGym, the failure rates are 2.04% for all three CASCADE variants (both nuclei), 14.41% for DetaNet (both nuclei), 0% for NMRNet (both nuclei), and 0.09%/0.59% for Mnova (¹H/¹³C, respectively). A common-subset comparison is not available because the published NMRGym results do not provide per-molecule predictions or validity masks for all baselines.
We additionally reproduced NMRNet using its five released checkpoints and compared it with NMR-LM on NMRGym. The MAE results are:
| Model | Nucleus | Mean MAE |
|---|---|---|
| NMR-LM | ¹H | 0.2237 ppm |
| NMRNet | ¹H | 0.1724 ppm |
| NMR-LM | ¹³C | 2.1925 ppm |
| NMRNet | ¹³C | 2.4306 ppm |
NMRNet has no failed predictions for either nucleus, while NMR-LM has a failure rate of 0.314% for ¹H and 0% for ¹³C. NMR-LM has higher ¹H MAE but lower ¹³C MAE than NMRNet. We have added the additional spectral prediction comparison to Appendix D.1.
Q6: On NMRSpec, do the NMRTrans/NMRMind numbers (taken from the original papers) use the molecular formula and the multiplicity/J-coupling information that NMR-LM receives (Appendix B.1)? Since you reproduced NMRTrans (Appendix D.1), please also report it on NMRGym.
Table 2 reports NMRTrans and NMRMind results from the original NMRTrans paper. The generative baselines were trained on the NMRSpec training split using their official implementations with molecular-formula conditioning. NMRTrans explicitly uses ¹H chemical shifts, integration, multiplicity, and J-couplings, together with ¹³C shifts. However, the paper does not explicitly specify whether NMRMind uses the same multiplicity and J-coupling information.
For our NMRTrans reproduction in Appendix D.1, we used the released checkpoint and official implementation for inference, without additional training. We did not report NMRTrans on NMRGym because NMRGym provides chemical-shift lists and molecular formulas but lacks the explicit multiplicity and J-coupling information required by the original NMRTrans input representation. A comparable evaluation on NMRGym would therefore require input adaptation and retraining.
Q7: How are acyclic molecules (empty Bemis–Murcko scaffold) handled by Exclude-Scaffold, and is Exclude-Molecule based on stereo-aware SMILES or InChIKey connectivity? Does exact match require correct stereochemistry?
For Exclude-Scaffold, none of the 27,054 NMRGym test molecules has an empty Bemis–Murcko scaffold, so this case does not arise in our evaluation. Exclude-Molecule uses RDKit canonical isomeric SMILES rather than InChIKey connectivity, preserving stereochemical distinctions in both NMRGym and NMRSpec. Exact-match accuracy also accounts for stereochemistry, so predictions with incorrect stereochemistry are counted as mismatches.
Q8: Please qualify "state-of-the-art" in the abstract as in the introduction ("among the evaluated generative models"), and state the absolute Top-1/Top-10 next to the relative gain.
Thank you for the suggestion. We have qualified the state-of-the-art claim in the abstract and added the absolute Top-1/Top-10 accuracies in the Introduction (line 88).
Response to Reviewer sAmq (Part 1/3)
We thank the reviewer for the detailed and thoughtful review. We have addressed the key concerns and provide our responses below.
Thank you for pointing out this asymmetry. Our original NMR-Solver reproduction followed its original FB-MO search procedure, applying molecular formula filtering only to the final outputs. To assess the impact of this difference, we further incorporated exact molecular formula constraints into NMR-Solver's fragment pairing and product validation. We evaluated both variants on the same 1,000 NMRGym test molecules under the Exclude-Scaffold protocol.
Internal formula constraints substantially improve NMR-Solver, confirming that the original asymmetry accounts for part of the performance gap. Nevertheless, NMR-CRAFT remains stronger across all Top-k metrics and substantially faster. We have added the formula-aware comparison and search-budget analysis to Appendix D.6.
Thank you for raising this concern. We followed the official NMR-Solver configuration (
use_H_split=False), consistent with its original evaluation protocol, which excludes multiplicity and J-coupling information. NMRGym provides chemical-shift lists without explicit splitting-pattern or J-coupling annotations. Thus, enabling multiplicity-aware scoring would require information unavailable in this benchmark.Thank you for these questions. Our original NMR-Solver reproduction follows its official protocol, with formula filtering applied after search. We additionally incorporated the exact formula into its fragment-based molecular optimization (FB-MO) and evaluated different search budgets on the same 1,000 NMRGym test molecules under Exclude-Scaffold.
R/P/C denotes the limits on retrieved candidates, fragment pairs, and candidate pool size, respectively. For NMR-Solver, R applies only to the initial retrieval, while the same P and C limits apply to each FB-MO iteration. Accuracy is reported in percent and time in seconds per molecule. Formula-aware search improves NMR-Solver, but NMR-CRAFT achieves a favorable accuracy–runtime trade-off. A complete Pareto analysis would require more extensive budget sweeps.
Both methods ran on the same four-A100 server. NMR-Solver used one GPU, while NMR-CRAFT originally used separate GPUs for sequential inverse and forward inference. Running both services on one GPU yielded 14.11 s/sample, comparable to the original 14.24 s/sample and approximately 14× faster than formula-aware NMR-Solver at its original budget.
As noted in W1-2, we retain the official
use_H_split=Falsesetting because NMRGym lacks explicit splitting-pattern and J-coupling annotations. We have clarified the evaluation configurations and added the budget comparisons to Appendix D.6.