- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces IonTrapBench, an offline benchmark for evaluating closed-loop mass-spectrometer tuning using recorded spectra, instrument configurations and fault annotations. It also proposes History-Gated Stopping, which uses historical fault rates to adjust stopping decisions. Experiments compare stopping policies under different history-access conditions and report tuning success, faulty stops, newly introduced faults and rescan costs, including a cross-analyte transfer evaluation.
Critical strengths
- The recorded spectra and configuration/replicate identifiers support controlled comparisons on the same initial cases, repair rules and scan budgets.
- Reporting success, faulty stopping, new faults and rescans shows consequences that window-level classification scores miss. The classifier comparison demonstrates that higher recall need not yield better tuning outcomes.
- Calibration, sequential and full-grid histories are evaluated separately, with full-grid access explicitly labeled as assistance. The held-out peak and cross-analyte experiments extend the evaluation beyond the classifier's training peaks.
Critical weaknesses
- The paper does not report the search range, objective or selected thresholds for the calibrated baseline in Table 2. It is unclear whether the search already includes a global fault-priority policy. Under Equation 4's default thresholds, HGS can change the ordinary action only when both the normal score and at least one fault score reach 0.5. At θ = 1, the no-history rule gives faults priority whenever the normal sigmoid score is below one. Without the selected thresholds, the endpoint's success–rescan trade-off cannot be compared with HGS.
- The reported gains could come from broader labeled-history coverage or more conservative stopping rather than matching risk to the current configuration. Full-grid assistance has more labeled history, while calibration-only HGS has success similar to calibrated and conformal baselines. There is no control that keeps the same history, coverage and risk-value distribution while removing the alignment between risk and the current configuration.
- Section 3.2 updates sequential history after each episode, using annotated reference faults. It does not specify which visited windows and replicate labels are added, or whether repeated observations are counted more than once. These choices affect both the estimated risk and history coverage.
- Equal cross-analyte success rates do not identify the remaining failures. Section 4.3 attributes them to the repair engine, including wrong adjustment directions. In Table 4, both fixed-threshold and HGS+PEG success are 98.0% overall and 95.7% on fault starts, while mean FN falls from 3.0 to 0, new faults rise from 0.6 to 1.4, and rescans rise from 2.82 to 2.95. These aggregate results could reflect unchanged repair failures, rescues offset by new failures, or faulty stops becoming deferrals or budget exhaustion. The paper needs paired terminal outcomes to distinguish these explanations.
Questions for authors
- Please specify the calibrated baseline's search range, objective and selected thresholds, including whether θ = 1 was considered. For that endpoint and the calibrated operating points, please report paired success/rescan results and the frequency of simultaneous positive normal/fault predictions.
- With the same historical records and coverage, how does HGS perform when risk values are shuffled among covered configurations? Please compare success, faulty stopping, new faults and rescans across calibrated operating points.
- Which visited windows and reference labels enter sequential history after an episode? How are repeated observations handled, including spectra incorrectly accepted as normal?
- For the same initial cases and replicate IDs, how many faulty-STOP episodes become successes, deferrals or budget exhaustion under HGS+PEG, and how many previously successful episodes become failures? Which paired outcomes support wrong adjustment directions as the remaining bottleneck?