- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes SPARC, which adapts a pretrained MS/MS predictor (Iceberg) to a target domain one query at a time. For each query, it retrieves 64 adduct- and instrument-matched labeled spectra based on fingerprint similarity and uses them to fine-tune the intensity model rather than using the query’s measured spectrum. The fragment generator is frozen, and the intensity model is updated with a similarity-weighted reconstruction loss and a teacher-student consistency loss. An EMA teacher, stochastic restoration, and rollback are used to limit model drift. The experiments cover adduct specialization, cross-library transfer, within-domain refinement, and five GNPS libraries.
Critical strengths
SPARC aims to adapt a pretrained MS/MS predictor to a target dataset at the prediction time. It shows improvement from the pretrained predictor without adaption.
Critical weaknesses
My main concern is that the experiments do not clearly separate the benefit of SPARC from simply training or fine-tuning Iceberg on the same labeled spectra available to SPARC. In the GNPS experiments, these spectra are also part of Iceberg’s original training data, while on NPLIB1, the only standard fine-tuning baseline performs worse than training from scratch. Moreover, although SPARC substantially improves over the pretrained model in the transfer settings, its margins over models trained on the target data are generally small, as are its gains within domain and on several GNPS libraries. These results are also reported without variance estimates, making it difficult to assess their robustness. Overall, the improvements appear to come primarily from increased peak precision rather than a consistent improvement across evaluation metrics.
Weakness 1: The baselines do not adequately isolate the benefit of SPARC, and several appear under-tuned.
SPARC has access to labeled target spectra in every experiment, but the comparisons do not consistently control for this advantage. For MassSpecGym and NPLIB1, the support library is the target training split, while validation labels are used for checkpoint selection (Sec. 4). For GNPS, SPARC uses the MSG training split, which is also the data used to train the Iceberg baseline in Table 4. Thus, SPARC is effectively performing additional target-data fitting, rather than adapting with information unavailable to the baseline.
The within-domain NPLIB1 result illustrates this issue. Re-fitting Iceberg on NPLIB1 increases CosSim from 0.586 to 0.616 (Table 2), even though there is no domain shift. This suggests that part of SPARC’s improvement may come from local per-query fitting, similar to test-time training on nearest neighbors [1], which is not evaluated.
The margins over models trained directly on target data are also modest. On MSG to MSG-MNa, SPARC(val) improves over Iceberg trained from scratch by only 0.022 CosSim (0.370 vs. 0.348; Table 1), while removing retrieval reduces SPARC to 0.344 (Table 5). On NPLIB1, retrieval-weighted fine-tuning alone reaches 0.587, essentially matching Iceberg trained on NPLIB1 (0.586; Sec. 5.2). Meanwhile, global fine-tuning reaches only 0.5545, below training from scratch, raising questions about whether this baseline was adequately tuned. NEIMS and GrAFF-MS also perform substantially worse than Iceberg, which makes those comparisons difficult to interpret.
Two simple baselines are particularly important: Iceberg pretrained on the union of the available libraries, with and without standard fine-tuning on the target split, and a similarity-weighted average of the 64 retrieved spectra. TENT and CoTTA use no labels, so comparisons with them do not isolate the contribution of retrieval.
Weakness 2: The improvements are small, inconsistent across metrics, and reported without variance estimates.
Several reported gains are quite small. On MSG M+H, SPARC(val) increases CosSim from 0.529 to 0.532, while MSE worsens from 0.853 to 0.907 (Table 2). On GNPS, the CosSim gain is only 0.005 for 3-Hydroxy Acyl Amides, and SPARC improves all three metrics on only two of five libraries (Table 4 and Sec. 5.4). The online TTT stage contributes only 0.006–0.020 CosSim (Tables 1 and 2).
No results are reported across multiple seeds or with standard deviations. This is important because SPARC is stochastic and order-dependent (Sec. 6), making it difficult to determine whether the 0.006–0.031 differences between ablations in Table 5 are robust.
There is also a substantial precision-recall trade-off. Peak recall drops from 0.550 to 0.449 on M+H and from 0.159 to 0.097 on M+Na (Table 3), but this is not discussed despite the potential impact of missing peaks on downstream identification.
Finally, the evaluation protocol makes comparison with prior work difficult. The paper retains only the top 100 predicted bins, evaluates only spectra that all methods can predict, filters support spectra by scaffold, and splits MassSpecGym by adduct (Sec. 4). The selection of the five GNPS libraries is also not explained. Results under the official MassSpecGym protocol and a downstream molecule-retrieval experiment would provide stronger evidence that the reported spectral improvements translate into better identification.
Weakness 3: The methodological novelty and generality are limited, and the computational cost is unclear.
Several components build directly on existing methods. The EMA teacher and stochastic restoration come from CoTTA, while retrieval-weighted fine-tuning is a form of local learning (Sec. 3.4). The main new component, support-calibrated bin-wise weighting (Eqs. 5–6), contributes 0.024 CosSim in one unreplicated experiment (Table 5).
SPARC is evaluated only with Iceberg, so its applicability to other pretrained MS/MS predictors remains untested. Because the fragment generator is frozen, SPARC also cannot recover diagnostic fragments missing from the pretrained model (Sec. 6). The entropy guard further favors sharper spectra by rejecting updates that increase entropy, which seems somewhat inconsistent with the paper’s argument that lower entropy does not necessarily indicate better fragmentation.
The computational overhead is also important but unreported. Each query requires processing 64 retrieved spectra with forward and backward passes, rather than a single forward pass. This could be substantial for large-scale in-silico library generation. In addition, the model state carries across queries, so predictions depend on query order (Sec. 6). The paper should quantify both the computational cost and the magnitude of this order dependence.
References
[1] Hardt, Moritz, and Yu Sun. "Test-time training on nearest neighbors for large language models." International Conference on Learning Representations. Vol. 2024. 2024.
Questions for authors
How does SPCAC compare with the test-time training on nearest neighbors [1] or the direct fine-tuning of iceberg on the target dataset? How does SPARC perform on the official MassSpecGym protocol? What is the contribution of the contribution of retrieval to the improvement by SPARC?
[1] Hardt, Moritz, and Yu Sun. "Test-time training on nearest neighbors for large language models." International Conference on Learning Representations. Vol. 2024. 2024.