- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes UncertFP, an uncertainty-aware multi-view evidence fusion framework for molecular fingerprint prediction from MS/MS spectra. The authors construct three spectral views and employ an evidence head to estimate bit-wise evidence strength and reliability scores. These are used to determine view-specific fusion weights, while a learnable coefficient controls evidence accumulation and a disagreement discounting mechanism reduces conflicting evidence. The fused evidence is then used to generate fingerprint predictions and uncertainty estimates through Beta distributions. Experiments across three datasets, two encoders, and three fingerprint dimensionalities demonstrate consistent improvements in fingerprint prediction and candidate-structure ranking.
Critical strengths
-
Distinguishing prediction probability from evidence strength. UncertFP explicitly models bit-presence probability and evidence strength separately, allowing the model to distinguish predictions with similar probabilities but different levels of evidential support. This provides a useful framework for uncertainty estimation and reliability-aware multi-view fusion.
-
Extensive experimental validation. The authors conduct experiments across three datasets, two encoders, and three fingerprint dimensionalities, demonstrating consistent improvements over the baselines. Comprehensive ablation studies further support the effectiveness of the proposed components. Additionally, candidate-structure ranking and selective prediction experiments demonstrate the effectiveness of uncertainty estimates.
Critical weaknesses
-
Potentially unfair baseline comparisons. UncertFP and the baselines use different checkpoint selection criteria: the baselines are selected based on validation fingerprint cosine similarity, whereas UncertFP uses validation Soft Tanimoto. Moreover, UncertFP introduces additional trainable parameters through its evidence heads and fusion modules.
-
Insufficient validation of learned evidence strength and reliability. Both evidence strength and bit-wise reliability scores play central roles in the proposed framework, but the authors do not directly validate whether these learned quantities reflect their intended meanings. In particular, the paper lacks an analysis of whether learned reliability scores correlate with the actual predictive accuracy of individual views. Furthermore, the selective prediction experiments compare uncertainty mass, which receives explicit error-ranking supervision, against predictive entropy, which does not. This makes it difficult to determine whether the improvement comes from evidential modeling or the additional supervision.
Questions for authors
-
Could the authors provide comparisons with parameter-matched baselines using the same checkpoint selection criteria to clarify whether the improvements stem from the proposed methods rather than additional model capacity or differences in training settings?
-
Could the authors provide analyses demonstrating that the learned evidence strength and bit-wise reliability scores reflect their intended meanings?
6 comments
Response to Reviewer bGRE (2/4)
W2 & Q2: Intended Meaning of Evidence Strength
R2: Evidence strength is intended to measure how much support each view provides, so it should decrease when the spectra provide less reliable information. To verify this, we conduct experiments on the CANOPUS dataset with 4096-bit fingerprints, using MIST and DiffMS as the baselines, and perturb only the test spectra. Each perturbation is applied at ratios of 20% and 50%, defined as the number of removed, added or corrupted peaks relative to the number of original peaks. We report the Soft Tanimoto of UncertFP (higher is better ) and Uncertainty, its mean uncertainty mass, which rises as the evidence decreases.
- None denotes the unperturbed test spectra.
- Drop peaks removes randomly selected peaks to simulate missing fragments.
- Add noise peaks inserts low-intensity peaks annotated with random subformulae of the precursor formula to simulate noisy peaks.
- Corrupt subformulae replaces the subformula annotations of selected peaks with other subformulae of the precursor formula to simulate incorrect subformula annotations.
| UncertFP-MIST-4096 | UncertFP-DiffMS-4096 | ||||
|---|---|---|---|---|---|
| Perturbation | Ratio | Soft | Uncertainty | Soft | Uncertainty |
| None | - | 0.48 | 0.12 | 0.47 | 0.15 |
| Drop peaks | 20% | 0.45 | 0.15 | 0.43 | 0.19 |
| 50% | 0.39 | 0.23 | 0.38 | 0.25 | |
| Add noise peaks | 20% | 0.46 | 0.14 | 0.45 | 0.16 |
| 50% | 0.44 | 0.17 | 0.44 | 0.18 | |
| Corrupt subformulae | 20% | 0.43 | 0.18 | 0.42 | 0.22 |
| 50% | 0.36 | 0.29 | 0.35 | 0.28 |
It can be observed that, the mean uncertainty mass rises with the perturbation ratio and is higher in the settings with larger drops in Soft Tanimoto. The learned evidence therefore decreases when the spectra provide less reliable information, in line with the intended meaning of evidence strength.
Response to Reviewer bGRE (3/4)
W2 & Q2 (cont.): Intended Meaning of Bit-wise Reliability Scores
R3: Reliability scores are intended to indicate how reliable each view is on each fingerprint bit, so a view with a higher reliability score on a bit should predict that bit more accurately. To verify this, we test whether the learned reliability scores correlate with the actual predictive accuracy of individual views. Reliability diagrams [1] check whether a confidence score reflects accuracy by grouping predictions by the score and comparing the accuracy across groups. Following this practice, we group the view predictions of UncertFP by their reliability scores and compare the precision across groups. We conduct the analysis on the test set of the CANOPUS dataset with 4096-bit fingerprints, using MIST and DiffMS as the baselines.
The details of the experimental setup above are explained as follows:
- Reliability scores. For each spectrum in the test set, the evidence head of UncertFP maps the representation of each of the three views to one reliability score per fingerprint bit.
- Selected predictions. To compare only predictions with high and similar bit-presence probabilities, we take every view prediction with a bit-presence probability of at least 0.9, in which a view confidently predicts the bit to be present.
- Groups. A prediction is labeled Highest if the reliability score of its view is the largest of the three views on that bit and Lowest if it is the smallest. To check whether the reliability scores encode a fixed preference for one view, we form these groups over all views and within each of the fragment-peak (F), neutral-loss (N) and peak-difference (Δ) views.
- Precision. The precision of a group is the fraction of its predictions whose bit is present in the true fingerprint (higher is better ).
| UncertFP-MIST-4096 | UncertFP-DiffMS-4096 | ||
|---|---|---|---|
| View | Rank of reliability score | Precision | Precision |
| All views | Lowest | 0.72 | 0.70 |
| Highest | 0.84 | 0.85 | |
| F | Lowest | 0.74 | 0.72 |
| Highest | 0.86 | 0.86 | |
| N | Lowest | 0.71 | 0.70 |
| Highest | 0.84 | 0.85 | |
| Δ | Lowest | 0.70 | 0.69 |
| Highest | 0.82 | 0.83 |
It can be observed that the mean bit-presence probabilities of the groups differ by at most 0.02, so the differences in precision between the groups do not come from differences in bit-presence probability.
Furthermore, over all views, the predictions from the view with the highest reliability score on a bit have higher precision than those from the view with the lowest reliability score for both baselines. Since the bitwise reliability weights increase with the reliability score, the fused prediction favors the view that is more accurate on that bit.
Additionally, within each of the F, N and Δ views, the predictions with the highest reliability score also have higher precision than those with the lowest. This indicates that the reliability scores follow the accuracy of a view on each bit rather than encoding a fixed preference for one view.
Therefore, the learned reliability scores correlate with the actual predictive accuracy of individual views, which reflects their intended meaning.
Response to Reviewer bGRE (1/4)
We sincerely appreciate your valuable feedback.
W1 & Q1: Checkpoint Selection Criteria and Parameter-matched Baselines
R1: To rule out the effects of different checkpoint selection criteria and additional parameters, we conduct experiments on the CANOPUS dataset with 4096-bit fingerprints, using MIST and DiffMS as the baselines, with MIST-4096 and DiffMS-4096 as the baselines. We report Soft Tanimoto and Binary Tanimoto (higher is better ↑).
1) Checkpoint selection criteria. We compare the baselines and UncertFP under the same training budget and checkpoint-selection metric.
It can be observed that both MIST-4096 and DiffMS-4096 show a slight decline in performance after an additional 300 epochs of training. One possible explanation is that the extra training causes the models to overfit. By contrast, with the same additional 300 training epochs and Cosine as the checkpoint-selection metric, UncertFP achieves a significant performance improvement over both baselines. This indicates that additional training is not the primary factor for the performance gains, but rather the model design of UncertFP. In addition, UncertFP performs somewhat better when Soft Tanimoto is used instead of Cosine as the checkpoint-selection metric, but the gain is far smaller than that resulting from UncertFP’s novel design. Therefore, the performance improvement stems primarily from our UncertFP, not merely from additional training or a different checkpoint-selection metric.
2) Additional parameters. If the improvements came from increased model capacity, a baseline with the same parameters would perform close to UncertFP. We therefore compare UncertFP with a parameter-matched baseline that uses the same checkpoint-selection metric as UncertFP, and train all models for 300 additional epochs.
It can be observed that UncertFP-F consistently achieves higher scores than the baselines, while UncertFP outperforms UncertFP-F by an even greater margin. This indicates that the gains from increased capacity are smaller than those from the N and Δ views, which do not add parameters. Furthermore, the ablation studies in Section 5.2 and Appendix E of the paper also demonstrate that incorporating the N and Δ views yields greater improvements in both metrics than adding the evidence head.
Therefore, compared with parameter-matched baselines using the same checkpoint selection criteria, the improvements stem primarily from our proposed UncertFP rather than merely from additional model capacity or differences in training settings.