acceptodds
Under review as a conference paper at ICLR 2027

What Counts as a False Discovery in Peptide Sequencing?

Abstract

In peptide sequencing, matching observed fragments does not necessarily establish sequence identity, complicating the interpretation of false-discovery estimates. We investigate this problem using isotope measurements, designed peptide libraries and public sequencing and error-estimation workflows. Isotope traces and fragment matches leave some sequence differences unresolved, whereas synthesis constraints distinguish alternatives under explicit assumptions about library composition. The distinction between fragment support and sequence agreement persists after confidence-based selection. Among spectra with identities supported by both synthesis design and database search, a Casanovo selection at a nominal 5% error rate yields 3.31% disagreement under a fragment-coverage criterion but 11.15% under a modified-sequence criterion. Calibrating confidence against these two criteria also produces different sequence-disagreement rates when the fitted models are applied to other biological studies. Even with the error criterion fixed, observed disagreement depends on the population used for calibration and evaluation, and on whether spectrum matches or distinct peptides are counted. Together, these results show why error estimates must be tied to the sequence identity being claimed, the evidence supporting its assessment and the predictions ultimately reported.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
de novo peptide sequencing, false discovery rate, confidence calibration, mass spectrometry, sequence identity, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iwVQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is concerned with characterizing different types of error rates in the context of de novo sequencing from proteomics tandem mass spectrometry data. However, I could not figure out exactly what problem the paper aims to address.

Critical strengths

None

Critical weaknesses

This paper is quite difficult to follow. I have worked with proteomics MS/MS data for a long time, have thought a lot about database search and de novo sequencing, and have worked on methods for error estimation in this domain. Yet I still had trouble following the reasoning through the Introduction of your paper. I feel quite sure that a typical ICLR reader would not be able to follow it at all. The remainder of the paper is, if anything, more dense and difficult to follow. Paragraphs do seem to be organized around a leading topic sentence, but transitions between paragraphs are often unclear, and many concepts are never introduced. E.g., "For mixed spectra, the identity claim must specify the designated precursor or any co-isolated component." Or the sudden appearance of "the E. coli source" (line 112), which has not been introduced or described.

Questions for authors

I cannot boil this down to a few questions that would materially affect my assessment. I found so many parts of the exposition difficult to understand that enumerating them would not be a useful endeavor.

Reviewer_GVTuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks what a false discovery means in de novo peptide sequencing, where a predicted sequence that explains the observed fragment peaks need not be the generating peptide. Section 2 formalises the question: for an accepted set A(t), the realised false discovery proportion depends on the equivalence relation defining identity (sequence/site, elemental formula, ion coverage), the discovery unit (PSMs vs peptide representatives), and the population and controls used for selection. The experiments use three sequencers (Casanovo 5.2 Orbitrap, InstaNovo 1.2, pi-PrimeNovo) on E. coli (two acquisitions, 4,458 and 7,046 spectra), human (36,049 spectra), yeast (40,000), a 131,036-spectrum synthetic library with 2,190 supported-anchor subset, and a phosphotyrosine library. Section 4.2 holds the accepted set fixed and varies only the error definition: for Casanovo on supported anchors at a nominal 5% selection, ion90 disagreement is 3.31% while modified-sequence disagreement is 11.15%, with the ordered residue-composition alias accounted for separately (10.74%). On human spectra these are 1.56% and 7.45%. Section 4.2 also fits isotonic score calibrators to the same E. coli rows under two supervision criteria (ion90 and sequence/site agreement) and shows the choice persists in cross-study transfer: on human spectra, ion-supervised maps give 10.27-13.70% sequence disagreement for Casanovo while sequence-supervised maps give 3.20-4.27%, with the contrast reproducing across five source-partition rotations. Section 4.3 holds the error criterion fixed and shows that calibration source, population restriction (design anchors vs supported anchors), discovery unit (PSMs 324/1,043 vs peptide representatives, with 151 additional library-membership mismatches on the same selections), and decoy noise pool (within-library vs cohort-wide) each move the reported rate. The paper proposes no new estimator or decision rule and the discussion limits itself to the recommendation that error assessment be reported at the level of the complete selection procedure.

Critical strengths

1. The experimental design cleanly separates three confounded factors by holding each in turn fixed. Section 4.2 holds the accepted set fixed and varies only the error definition, which is the right way to show that the 3.31% / 11.15% gap is not a selection artefact but an interpretation gap: the same spectra, scores, and nominal threshold produce rates on opposite sides of the nominal 5% under two reasonable identity criteria. Section 4.2 then holds the model and fitting rows fixed and varies only the supervision criterion for calibration, with the supervision contrast reproducing across five partition rotations (Table 2: 10.27-13.70% ion-supervised vs 3.20-4.27% sequence-supervised sequence disagreement for Casanovo on human spectra). Section 4.3 holds the error criterion fixed and varies population, discovery unit, and control construction. This factor-at-a-time structure is unusually disciplined for a measurement paper and makes each finding attributable rather than mixed.

2. The identity evidence is built from genuinely orthogonal sources rather than one reference annotation. The "supported anchors" population (2,190 spectra, 440 backbones) requires simultaneous agreement from three independent instruments: synthesis design (which constrains the candidate identity set), fragment-search annotation (sequence agreement from a database tool), and isotope-zero precursor assignment. Figure 1 and Figure 2 further use measured isotope traces and site-determining ions to construct a direct physical argument that competing predictions can share all observable support, independent of any database-search reference. This matters because the paper's core claim is that the "reference" is doing most of the work in defining the error rate, and the credibility of that claim depends on being able to show cases where multiple independent instruments agree. The paper does this and reports negative controls (e.g., ordered residue-composition aliases that preserve the elemental formula but swap residue order) that would be invisible to a single-source reference.

Critical weaknesses

1. The paper is diagnostic without being prescriptive, which leaves the reader without an actionable framework. The experiments establish that different error criteria give different rates on the same accepted set, but the Discussion offers only "an error estimate is interpretable relative to the identity claim it evaluates" and "the appropriate endpoint follows the scientific question". For a measurement audit paper this is honest, but for a research contribution it leaves open the natural next question: given a scientific goal (modified-sequence assignment vs peptide inventory vs post-translational modification localisation), what is the recommended error definition, which calibration supervision should be used, and how should the error rate be reported such that two labs doing the same science can compare numbers? Without such defaults, two labs reading this paper will still produce incomparable FDR estimates on the same data. A unified reporting template, or a decision rule mapping scientific question to criterion-plus-unit-plus-population, would convert the diagnostic into a prescription.

2. Venue fit for ICLR is a legitimate concern that the paper does not strongly address. The primary area is "applications to physical sciences", which is a valid ICLR track, but the technical substance is specialised to mass-spectrometry proteomics (de novo peptide sequencers, target-decoy competition, isotope traces, Prosit features, Winnow, GraphNovo features). The connection to general ML (calibration under distribution shift) is drawn in the introduction but is not developed as a general-ML lesson: the paper does not argue, for instance, that this failure mode arises in other scientific-ML calibration settings where the "correctness event" has multiple reasonable operationalisations. For ICLR this risks being filed as a strong proteomics paper that would be a stronger fit at ISMB, RECOMB, or Bioinformatics. The paper would benefit from a half-page section drawing explicit parallels (e.g., medical imaging where "correct detection" has multiple operationalisations, or physics simulators where "correct trajectory" can be defined at several levels of resolution) with the question "how general is this measurement-definition problem".

Questions for authors
  1. Recommended defaults. Given a scientific question (modified-sequence identification, peptide-inventory estimation, PTM-site localisation), what error definition, discovery unit, and population restriction do you recommend as the default? A compact decision table would convert the paper's diagnostic conclusions into actionable guidance.

  2. Reference ambiguity. The 11.15% sequence-disagreement rate on supported anchors treats search-plus-design-plus-isotope agreement as ground truth. What is your best estimate of this reference's own error rate, and how does the headline contrast change under a lower bound and an upper bound on reference error?

  3. Training-sequence overlap. Appendix A.5 reports proxy sequence overlap but exact training-set membership is unknown for the released checkpoints. On yeast (lower proxy overlap), does the ion-supervised vs sequence-supervised gap in Table 2 narrow in a way that would be expected if proxy overlap were the main driver? If yes, how should calibration be reported for sequencers trained on undisclosed data?

  4. Generalisation beyond proteomics. Where else have you seen the measurement-definition problem bite in ML calibration (medical imaging, physics simulators, structure prediction)? A one-paragraph argument that this is a general scientific-ML phenomenon rather than a proteomics quirk would strengthen the paper's fit at ICLR.

  5. A unified metric. Have you considered reporting a vector of error rates (sequence, formula, ion90, library-membership) under a single selection, so that at least downstream readers can see the four numbers that this paper shows are jointly necessary? If so, what blocks this from being a standard output of de novo sequencers today?

Reviewer_H2RMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines how false discoveries should be defined in de novo peptide sequencing. The authors compare sequence identity, elemental composition, fragment recovery, and peptide presence using isotope-labeled data, biological samples, and synthetic peptide libraries. They show that the same accepted predictions can have substantially different disagreement rates under these criteria. For example, a nominal 5% Casanovo selection on supported anchors gives 11.15% sequence disagreement and 3.31% Ion90 disagreement. The paper also examines how calibration labels, evaluation populations, decoy construction, and discovery units affect the reported results. Its central recommendation is to specify the identity claim associated with an error estimate.

Critical strengths
  1. Comparing different correctness criteria on the same accepted predictions is a useful experimental design. Holding predictions, scores, and thresholds fixed makes Figure 3 straightforward to interpret and isolates the effect of changing the evaluation criterion.

  2. The synthetic-library examples illustrate the ambiguity in the spectral evidence. Figure 2 shows how different sequences can recover all observed reference peaks, while a companion charge state provides additional distinguishing evidence. These examples give the reader a concrete basis for understanding the aggregate comparisons.

Critical weaknesses
  1. The advance over prior work is insufficiently established. The introduction already identifies sequence ambiguity and alternative evaluation conventions as known issues. The experiments quantify differences between these conventions, and the calibration analysis shows that training against different correctness labels changes the resulting selections. The direction of these effects follows naturally from the setup. Their magnitude is useful to document, but the paper provides limited additional understanding of when such discrepancies become large or how to anticipate them in a new dataset. I find the contribution largely descriptive, which is the main reason I do not support acceptance.

  2. The synthetic results leave reference uncertainty and sequencing error difficult to separate. The full-population bounds assume that the generating peptide belongs to the enumerated candidate set, while supported anchors require agreement with a database-search annotation. Winnow disagreement changes from 13.77% on design anchors to 2.32% on supported anchors, but the populations differ in both reference support and spectral composition. The controls in Appendix C.4 address feature recomputation; they do not establish how much of the difference comes from incorrect design-only assignments versus genuinely easier spectra. Independent validation of a sample of the excluded disagreements is needed to resolve this distinction. This matters because it determines how much the synthetic results establish about physical sequencing errors and their prevalence.

Questions for authors
  1. Which finding provides the clearest advance over the sequence-ambiguity and evaluation-convention studies cited in the introduction? What additional conclusion does it establish beyond quantifying differences between correctness criteria?

  2. What evidence distinguishes incorrect design-only assignments from genuinely difficult sequencing cases among the spectra excluded from supported anchors? Can the authors estimate their respective contributions to the observed disagreement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.