- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper adapts a pretrained multimodal structure-elucidation generator (the Multitasking model of Alberts et al., 2025, trained on simulated formula + ¹H NMR + ¹³C NMR + MS + IR) to experimental spectra without target structure labels. On 3,669 SDBS molecules the unadapted model reaches 50.70% Top-1 from ¹³C alone but only 5.94% with all spectra, which the authors call negative transfer. SMART (i) trains a router on perturbed source examples to pick, per target molecule, a spectral subset whose frozen-model prediction serves as teacher; (ii) adapts a full-spectrum student on the unlabeled target cohort with encoder InfoNCE plus shared-prefix decoder Jensen–Shannon alignment to the teacher (Stage 1), with a controller that chooses checkpoints from prediction drift and formula feasibility; and (iii) keeps predictions that are identical, valid and formula-consistent across selected checkpoints as pseudo-labels and fine-tunes on random spectral subsets (Stage 2). The reported full-spectrum Top-1 rises to 55.66% (4.96 points above zero-shot ¹³C and 3.41 above adapted ¹³C); on 1,995 Chemotion records it rises from 0.100% to 4.511%. The paper also reports an objective ablation (Fig. 5), structure-quality metrics on a separate 500-record cohort, attention analyses and a Blackwell-style argument that additional observations cannot increase Bayes risk.
Critical strengths
-
The paper documents a real and practically relevant failure mode. Table 1 shows that a generator trained on simulated multimodal spectra degrades from 50.70% (¹³C only) to 5.94% Top-1 when all experimental spectra are supplied, and Appendix D quantifies modality-specific mismatches between simulated and experimental inputs (e.g., proton peak splitting with factor 4.36, MS precision of 0.20 with about five times as many peaks). This diagnosis is useful for anyone deploying simulation-trained spectral models.
-
Label-free adaptation on the target cohort recovers most of the lost full-spectrum accuracy. Taking the reported numbers at face value, full-spectrum Top-1 goes from 5.94% to 55.66% and Top-10 from 14.91% to 80.97% without using target structures for training, and code is said to be available. However, see the weaknesses for why I cannot attribute this result to the proposed components.
Critical weaknesses
1. Several reported figures and numbers are transformed or drawn from different runs, and the main-text claims rely on them without disclosure.
- Figure 14 (checkpoint interval). Appendix G.2 (l. 1665–1667) states that the graphic anchors interval three to 2,042/2,807/2,956 hits and places interval four five hits lower, "changing the original ranking". According to Appendix B.5 (l. 916–917), the original Top-1 values are 55.41% for interval three and 55.74% for interval four. 2,042/3,669 = 55.66%, the paper's headline number. The figure therefore shows interval three raised to the headline value and interval four below it, the opposite of the measured ordering.
- Figure 5 (objective ablation). Appendix G.2 (l. 1668–1671) states that a "KL/InfoNCE pilot" with 128 records and one epoch had raw Top-1/5/10 maxima of 13.28125/21.09375/25.0%, and that "display offsets" of 42.37875/55.41625/55.57000 points were added so that they appear as 55.66/76.51/80.57%. Appendix G.1 (l. 1654–1655) says the "hybrid-objective and motivation graphics" use exactly 55.66/76.51/80.57%. Section 5.3 (l. 404–407) and Appendix B.7 (l. 989–995) present the Figure 5 values (InfoNCE 14.9, JS 54.5, KL 54.1, CE 51.4, hybrid 55.66/76.51/80.57) as the SDBS objective comparison and conclude that the hybrid improves over JS by 1.16/2.31/1.77 points; the caption of Figure 5 (l. 400–401) mentions neither a pilot nor offsets. The KL value of 54.1 equals 15/128 + 42.37875, consistent with this reading. Unless I misread Appendix G.2, the objective ablation is not a measurement on the 3,669-molecule cohort.
- Rows and figures from different runs. The H+C+MS row of Table 1 (55.41/76.12/80.54) is identical to the Full-view endpoint of a separate "pseudo-CE, interval-3 checkpoint" reported in Appendix G.1 (l. 1659–1660). The attention analysis uses a "historical" checkpoint (Fig. 7), and the layerwise attention values are "transcribed from figure summaries" (l. 1660–1661). The 500-record structure-quality study has a "distinct checkpoint lineage" (l. 362) and its full-spectrum ECFP4 Tanimoto after adaptation is 0.0665 (Table 5), which is incompatible with a model whose exact-match accuracy is anywhere near 55%; yet l. 441–444 cite it as evidence for SMART.
- All experiments use a single seed (3247, Table 9) and the "±" values in Table 1 are not defined. In addition, the running header reads "ICLR-format working manuscript / SMART / EN / N9-RR" and the author block "Independent working draft" rather than the ICLR 2027 template text.
2. None of the three claimed components is supported by a controlled comparison, and the simplest alternatives are not evaluated.
- Router. It selects ¹³C NMR for 97.5% of molecules (Appendix A.2), and the routed teacher reaches 48.87% Top-1 (l. 424, 458), below the 50.70% of always using ¹³C. The authors themselves write that a fixed-carbon control is essential (l. 636–637) but do not report it.
- Hybrid InfoNCE + JS alignment. The only evidence is the +1.16-point difference in Figure 5, which is affected by the issue above; the paper acknowledges that matched repeated runs would be needed (l. 418–419).
- Dynamic checkpoint controller. Fixed intervals give 54.70–55.74% Top-1, the best of which exceeds SMART's 55.66% (Appendix B.5), and the paper states that these records do not establish an advantage over matched fixed schedules (l. 463–464).
- Appendix F.1–F.4 (l. 1588–1639) describes the needed controls — fixed-carbon inference, router-only inference, fixed-carbon-teacher adaptation, per-molecule transition analysis, repeated seeds, paired intervals — as protocols that "should" be run, without results.
- Missing baselines. No standard test-time adaptation baseline (Tent, EATA, SAR, or plain pseudo-label self-training from a fixed ¹³C teacher) is compared, although these are cited (l. 095–096). Appendix D attributes the negative transfer largely to input mismatches (proton multiplets split into more lines, MS spectra with about 5× more peaks), so simple preprocessing (merging multiplets, keeping the top-k MS peaks) is an obvious baseline. The router's perturbation profiles are calibrated from 2,669 real SDBS train/validation records and 1,059 same-molecule real–simulated pairs per modality (Appendix D.1, l. 1357–1362), i.e., labelled target-domain data; supervised fine-tuning on these records is the most direct comparison and is not reported, and it is not stated whether these records overlap with the 3,669 evaluation molecules. This also qualifies the "zero-shot" and "without target structural labels" framing.
Questions for authors
Clarifications that could change my assessment
- Please list every figure and table whose plotted or tabulated values were transformed (offsets, anchoring, rescaling) or taken from a run other than the 3,669-molecule InfoNCE–JS pipeline, and provide the raw values. In particular: is Figure 5 the 128-record pilot described in Appendix G.2, and why does its caption not say so?
- Why is the H+C+MS row of Table 1 identical to the Full-view endpoint of the pseudo-CE interval-3 checkpoint in Appendix G.1?
- Please report fixed-carbon-teacher adaptation (same Stage 1 and Stage 2, router removed), router-only inference, and at least one standard TTA baseline, all with several seeds and paired confidence intervals.
- What do the "±" values in Table 1 represent, given that all runs use seed 3247?
- Are the 2,669 real SDBS records used to calibrate the perturbation profiles disjoint from the 3,669 evaluation molecules? What does supervised fine-tuning of the base model on these records achieve?
- What preprocessing is applied to experimental ¹H and MS inputs, and how does the unadapted model perform after merging multiplets and keeping the top-k MS peaks by intensity?
Suggestions (not required for my assessment)
- Explain why the 500-record cohort's ECFP4 similarity (0.0665 after adaptation) is so low, and evaluate structure quality on the same model and cohort as the main results.
- Describe how the "strict v8" Chemotion cohort was selected and whether earlier cohort versions were evaluated.