acceptodds
Under review as a conference paper at ICLR 2027

When Better Classifiers Make Worse Tuners: IonTrapBench and History-Gated Stopping for Closed-Loop Mass Spectrometer Tuning

Abstract

In autonomous instrument tuning systems based on learned classifiers, spectral diagnoses determine when to stop or continue adjustment. However, classification errors not only affect the assessment of the current signal but may also change subsequent observations through parameter updates. Window-level classification metrics therefore cannot directly substitute for evaluation of the complete tuning process. We reveal this prediction–decision misalignment in autonomous mass spectrometer tuning: retraining that improves window-level fault recall reduces the tuning success rate; false negatives lead to premature stopping, whereas false positives may introduce new faults through unnecessary parameter adjustments. To systematically investigate this problem, we propose IonTrapBench, which extends conventional window-level fault classification evaluation to a trajectory-level closed-loop decision evaluation paradigm that includes parameter states, action consequences, and final outcomes. On this basis, we propose History-Gated Stopping, which uses historical fault risk at a parameter state to adjust stopping decisions, combining predictions with state-level decision risk without retraining the classifier. Under sequential history, the method reduces false negatives by 88.2%; with full-history assistance, it increases the average tuning success rate from 84.2% to 91.6%. We plan to release the data and evaluation materials to support joint research on spectral diagnosis, parameter adjustment, and their final outcomes.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Autonomous instrument tuning, Mass spectrometry, Closed-loop tuning, History-Gated stopping, Spectral fault diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7zkGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces IonTrapBench, an offline benchmark for evaluating closed-loop mass-spectrometer tuning using recorded spectra, instrument configurations and fault annotations. It also proposes History-Gated Stopping, which uses historical fault rates to adjust stopping decisions. Experiments compare stopping policies under different history-access conditions and report tuning success, faulty stops, newly introduced faults and rescan costs, including a cross-analyte transfer evaluation.

Critical strengths
  • The recorded spectra and configuration/replicate identifiers support controlled comparisons on the same initial cases, repair rules and scan budgets.
  • Reporting success, faulty stopping, new faults and rescans shows consequences that window-level classification scores miss. The classifier comparison demonstrates that higher recall need not yield better tuning outcomes.
  • Calibration, sequential and full-grid histories are evaluated separately, with full-grid access explicitly labeled as assistance. The held-out peak and cross-analyte experiments extend the evaluation beyond the classifier's training peaks.
Critical weaknesses
  • The paper does not report the search range, objective or selected thresholds for the calibrated baseline in Table 2. It is unclear whether the search already includes a global fault-priority policy. Under Equation 4's default thresholds, HGS can change the ordinary action only when both the normal score and at least one fault score reach 0.5. At θ = 1, the no-history rule gives faults priority whenever the normal sigmoid score is below one. Without the selected thresholds, the endpoint's success–rescan trade-off cannot be compared with HGS.
  • The reported gains could come from broader labeled-history coverage or more conservative stopping rather than matching risk to the current configuration. Full-grid assistance has more labeled history, while calibration-only HGS has success similar to calibrated and conformal baselines. There is no control that keeps the same history, coverage and risk-value distribution while removing the alignment between risk and the current configuration.
  • Section 3.2 updates sequential history after each episode, using annotated reference faults. It does not specify which visited windows and replicate labels are added, or whether repeated observations are counted more than once. These choices affect both the estimated risk and history coverage.
  • Equal cross-analyte success rates do not identify the remaining failures. Section 4.3 attributes them to the repair engine, including wrong adjustment directions. In Table 4, both fixed-threshold and HGS+PEG success are 98.0% overall and 95.7% on fault starts, while mean FN falls from 3.0 to 0, new faults rise from 0.6 to 1.4, and rescans rise from 2.82 to 2.95. These aggregate results could reflect unchanged repair failures, rescues offset by new failures, or faulty stops becoming deferrals or budget exhaustion. The paper needs paired terminal outcomes to distinguish these explanations.
Questions for authors
  1. Please specify the calibrated baseline's search range, objective and selected thresholds, including whether θ = 1 was considered. For that endpoint and the calibrated operating points, please report paired success/rescan results and the frequency of simultaneous positive normal/fault predictions.
  2. With the same historical records and coverage, how does HGS perform when risk values are shuffled among covered configurations? Please compare success, faulty stopping, new faults and rescans across calibrated operating points.
  3. Which visited windows and reference labels enter sequential history after an episode? How are repeated observations handled, including spectra incorrectly accepted as normal?
  4. For the same initial cases and replicate IDs, how many faulty-STOP episodes become successes, deferrals or budget exhaustion under HGS+PEG, and how many previously successful episodes become failures? Which paired outcomes support wrong adjustment directions as the remaining bottleneck?
Reviewer_i4AFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how spectral classification errors affect closed-loop mass spectrometer tuning. It introduces IonTrapBench, an offline replay benchmark that connects measured spectra with instrument configurations and fault annotations. The authors show that higher fault recall does not consistently lead to better tuning outcomes and propose History-Gated Stopping (HGS), which uses the historical fault rate of the current configuration to adjust the evidence required for stopping without retraining the classifier. The evaluation considers different classifier checkpoints and levels of access to history, together with history transfer between analytes.

Critical strengths
  1. The motivation is clear and addresses a practical issue in autonomous instrument control. The paper explains how classification errors can affect later states through stopping and parameter adjustments. The classifier comparisons provide a concrete example of why window-level recall alone is insufficient for evaluating a tuning system.
  2. The replay benchmark supports controlled comparisons of stopping policies. It links measured spectra to parameter configurations and fault annotations, while fixing initial conditions, observation identifiers, repair rules, and scan budgets across policies. Reporting premature stops and newly introduced faults alongside final success and rescans helps explain how each policy changes the tuning process.
  3. HGS is simple and requires no classifier retraining. I appreciate that the authors distinguish different levels of access to history and acknowledge the costs of more conservative stopping. The cross-analyte experiments also examine whether existing instrument history remains useful when the sample changes, while recognizing that better stopping decisions cannot resolve failures caused by the repair policy.
Critical weaknesses

The contribution of the gating design is not yet fully isolated from the benefit of access to historical information. The strongest result uses history covering the recorded parameter grid, while the calibrated and conformal baselines rely on calibration data. Although the paper explicitly identifies full history as an assisted condition, a simpler stopping rule with access to the same records would make the contribution of HGS clearer. The construction of the full-history pool also needs a more precise account of its relationship to replay observations. 2. The practical benefit under limited history needs a clearer assessment of scanning cost. Sequential history reduces premature stopping but requires more rescans, and the gains in final success are modest. Calibration-only HGS performs similarly to calibrated thresholding. The paper acknowledges this trade-off and reports some operating points with similar rescan counts, but an explicit cost-matched comparison or threshold sweep would help establish when sequential history is worth the additional scans. 3. The role of the LLM remains unclear across the method description and appendix. The main text specifies a fixed STOP/ADJUST rule, Appendix A.1 describes the LLM as selecting this action, and Appendix C asks it to propose parameter adjustments. The paper provides the prompt and a caching mechanism, but it does not clearly explain how these components interact in each reported experiment. This affects both the interpretation of the results and the reproducibility of the control procedure.

Questions for authors
  1. How much of the improvement comes from the gating rule itself, given the same historical information? A baseline using the same records and repair rules but a simpler stopping criterion would be useful. Please also clarify whether the precomputed history excludes the specific spectra or observation IDs used during replay.
  2. Can the authors compare HGS, calibrated thresholding, and conformal stopping at matched average rescan counts? Curves showing final success, premature stopping, and scanning cost across operating thresholds would help assess the practical trade-off.
  3. Could the authors provide a unified control flow showing which component selects STOP/ADJUST, proposes parameter updates, and executes the final grid move? Would a direct implementation of the stated rules reproduce the results? As a minor reproducibility clarification, please also reconcile the parameter ranges in Table 1 and Appendix C and explain whether the appendix describes a restricted replay grid.
Reviewer_FmtdOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  1. Insufficient Research Motivation and Innovation Positioning Figure 1 provides a conceptual illustration of the discrepancy between classifier predictions and closed-loop tuning outcomes, but it does not clearly identify the specific conditions under which existing methods exhibit critical limitations or explain how the proposed method addresses these limitations. Furthermore, the use of historical information to assist parameter adjustment does not, by itself, constitute a clear algorithmic innovation. The manuscript does not adequately distinguish HGS from existing approaches to history-aware decision-making, threshold adjustment, and closed-loop tuning. Consequently, the central research motivation and the necessity of the proposed innovation remain unconvincing.
  2. Limited Methodological Innovation and a Relatively Simple Core Mechanism The core mechanism of HGS primarily applies a multiplicative correction to the classifier's normal score using historical failure rates and then uses threshold-based rules to select STOP or ADJUST. The overall design resembles a heuristic score-adjustment strategy and does not demonstrate a substantive algorithmic breakthrough. Figure 3 also fails to clearly present the technical interfaces, data flow, and decision logic among the different modules. This limitation obscures the substantive advantages of the proposed method over conventional score correction and threshold-based decision-making. The authors are advised to further justify the necessity of the design and validate its independent contribution through targeted comparative and ablation experiments.
  3. Unclear System Execution Logic and Implementation Details Equation (4) in the main text uses deterministic threshold rules to determine STOP or ADJUST, whereas the appendix links action selection to an LLM. However, the prompt primarily requests a reasoning process and parameter-adjustment recommendations, and the manuscript does not clearly specify whether the LLM actually makes the stopping decision. In addition, Table 1 and Appendix C.3 provide inconsistent parameter bounds for injection time and scan time. These issues prevent an accurate assessment of the system's actual execution workflow and experimental implementation. The authors are advised to clarify the responsibilities of each module and the execution order, as well as verify the consistency between the specified parameter rules and the actual implementation.
  4. Insufficient Experimental Fairness and Attribution of Performance Gains The most substantial performance improvement in Table 2 occurs under the Full History condition. However, the current comparisons do not adequately distinguish the benefits of historical-information coverage from those of the HGS-specific decision mechanism. When different methods use different historical-information conditions, the source of the performance improvement remains unclear. The authors are advised to conduct more comprehensive comparisons under identical historical-information conditions and experimental budgets to establish the practical advantages of HGS over other reasonable decision-making strategies.
  5. Insufficient Experimental Support for the Main Conclusions and Cross-Substance Validation The paper uses three MLP models to demonstrate the discrepancy between classifier performance and closed-loop tuning outcomes. However, the experimental scale and model diversity do not adequately establish the generality of this phenomenon. Meanwhile, although the cross-substance experiment in Table 4 reduces FN from 3.0 to 0, the final success rate remains 98.0%, while the numbers of new failures and rescans increase. The current results do not adequately establish that HGS consistently improves final tuning outcomes. The authors are advised to conduct more comprehensive paired trajectory analyses and clarify the relationship between the reduction in erroneous stopping decisions and the improvement in final tuning performance.
  6. Insufficient Validation of Historical Risk Estimation and Stopping-Rule Reliability When fewer than three exact-match historical records are available, HGS expands the query to parameter configurations one step away from the target configuration. However, neighboring configurations do not necessarily exhibit identical failure risks, and the current experiments do not adequately characterize the effects of this strategy on risk estimation and final decisions. In addition, Equation (4) still permits stopping when the corrected normal score and all failure scores fall below their respective thresholds. The rationale for this rule under conditions of high historical risk requires further validation. The authors are advised to analyze the reliability of risk estimation under sparse historical data and evaluate the potential for erroneous stopping or unnecessary adjustment through specific parameter-tuning trajectories.
Critical strengths
  1. Practical Significance of the Research Problem The paper examines the effects of classifier prediction errors on subsequent control actions and final parameter-tuning outcomes. This research direction has practical relevance.
  2. Potential Value of the Evaluation Benchmark IonTrapBench integrates spectral classification with the evaluation of complete parameter-tuning trajectories. This benchmark provides an experimental basis for analyzing the effects of stopping decisions on final control outcomes.
Critical weaknesses
  1. Insufficient Justification of Methodological Innovation The core HGS algorithm remains relatively simple, and the manuscript does not adequately establish its independent advantages over other history-aware decision-making methods. The research motivation and the distinctions between HGS and related approaches also remain unclear.
  2. Insufficient Rigor in Method Description and Technical Implementation The main method figure does not clearly convey the algorithmic mechanism. The system decision workflow contains unresolved ambiguities, and some parameter-rule descriptions remain inconsistent. These issues impair the interpretability and reproducibility of the proposed method.
  3. Insufficient Experimental Evidence for the Main Conclusions The most substantial performance gain depends on complete historical information. The experiments validating the central phenomenon remain limited in scale, and the cross-substance experiments do not demonstrate an improvement in the final success rate. The current validation also provides insufficient evidence regarding the reliability of historical risk estimation and the applicability conditions of the stopping rule.
Questions for authors

no

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.