acceptodds
Under review as a conference paper at ICLR 2027

Validity by Design: The Price of Safe Early Stopping in Closed-Loop Battery Testing

Abstract

Finding longer-lived lithium-ion batteries means cycling candidate cells until they wear out, weeks to months of channel time each. A campaign can save that time by choosing each next candidate from earlier results and stopping unpromising tests early, but a stop is irreversible and destroys the evidence that a design would have succeeded. Conformal prediction bounds the rate of such false stops in finite samples when the calibration cells are exchangeable with the cell being judged, and Bayesian optimization breaks that exchangeability. We rebuild it by design with the ε-stream, which sends each freed channel slot to a calibration cell with probability ε, drawn from a pre-registered sequence the optimizer never reads, so the conformal weights are exactly known and the certified modes carry finite-sample guarantees for the rule as deployed. We prove a ladder of such guarantees, each priced in channel time, calibration cells or delay, together with two classical floors and one impossibility, since on calibration pools holding a fixed number of cells per design those weights admit no distribution-free guarantee exact at every level. Replaying five public battery corpora, we find that successive halving, a heuristic from prior battery-testing frameworks, falsely stops as many as 96% of good cells, while our audit-only weighted rule stays within α on every seed of the unbudgeted replays; the certified modes' covered first stops reach about 1% of off-spec cells, so at these calibration sizes a certificate costs more channel time than early stopping saves.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, distribution-free uncertainty quantification, early stopping, risk control, Bayesian optimization, sequential experimental design, battery degradation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9b42OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Stopping a battery test early can save considerable time, but an incorrect decision also prevents the experiment from establishing whether a candidate design would have met its lifetime target. The central problem addressed here is how to control such false stops when subsequent experiments are selected adaptively using earlier results.

The proposed framework combines a frozen lifetime predictor, conformal calibration using known-good cells, and a separate ε\varepsilon-stream that collects calibration data independently of the acquisition process. The aim is to obtain known sampling probabilities and establish finite-sample guarantees for the stopping rule. The theoretical results cover different settings, including per-loaded-slot false-stop control across all looks under block-exclusive calibration and first-stop control under shared or front-loaded calibration, subject to the stated coupling qualifications, together with analyses of calibration granularity, sample reuse, and resource costs.

Experiments on five public battery datasets, together with additional case studies, show that aggressive stopping can eliminate many good cells. However, the block-exclusive configuration associated with the strongest all-decision guarantee does not stop in the reported replays, while the more practical weighted historical-pool rule remains audit-only. The authors conclude that the cost of stronger certification can exceed the testing time saved under the evaluated conditions.

Critical strengths

1. The paper addresses a meaningful consequence of early stopping that standard efficiency measures can overlook.

A prematurely stopped battery test cannot simply be resumed, and stopping a good cell can mean losing the evidence needed to identify a successful design. The paper makes this distinction concrete by measuring false stops and successful observed endpoints, rather than treating every reduction in testing time as a benefit. The replay experiments also show why this matters: aggressive halving-based stopping can discard a large fraction of good cells. This is a useful problem formulation for expensive physical experiments.

2. The paper provides concrete examples of where statistical guarantees do and do not apply.

Proposition 6 gives an explicit counterexample showing that the proposed pooled conformal statistic need not be valid under fixed stratified calibration, even when the individual importance weights are known. The analysis also distinguishes guarantees for fixed physical cells from those for adaptively selected loading slots, and makes clear that first-stop control is not the same as controlling every subsequent stop. These are useful technical distinctions. Although I question the overall significance of the new results, they provide a substantive basis for the work.

Critical weaknesses

1. The theoretical results do not yet establish a sufficiently important advance over the closest prior work.

The paper raises a reasonable question, but several of the statistical difficulties at its center have already been studied. Wieczorek (2023) explains why standard covariate-shift conformal arguments do not directly carry over to stratified sampling. Tibshirani et al. (2019) establish weighted conformal prediction under distribution shift, while Fannjiang et al. (2022) extend the analysis to feedback-driven data selection. Calibrated false-stop control for expensive experiments has also been investigated by Xiao et al. (2026), although that work does not address the same closed-loop setting.

There are new results here, particularly Proposition 6, which gives an explicit counterexample for the pooled statistic used in the paper. However, its conclusion concerns that statistic under particular sampling designs; it does not rule out other valid approaches. Likewise, some of the reported limitations follow from established statistical bounds, while others depend on the proposed certification schemes. For example, Theorem 4's quadratic calibration cost applies to a particular class of private-block procedures certified through a union bound, rather than to repeated safe stopping in general.

The authors do distinguish these different kinds of limitations, which I appreciate. My concern remains that, once those qualifications and the closest prior work are taken into account, the central new insight is less substantial than the paper's emphasis on the "price of validity" suggests. I am not convinced that the analysis establishes a sufficiently important new limit or design principle for safe closed-loop stopping.

References

Clara Fannjiang, Stephen Bates, Anastasios N. Angelopoulos, Jennifer Listgarten, and Michael I. Jordan. Conformal prediction under feedback covariate shift for biomolecular design. Proceedings of the National Academy of Sciences, 119(43), 2022.

Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems, volume 32, pp. 2530–2540, 2019.

Jerzy Wieczorek. Design-based conformal prediction. Survey Methodology, 49(2):443–473, 2023.

Yongrui Xiao, Mengjia Zhu, Yu Zhang, Dimitrios Tsaoulidis, and Tao Chen. Confidence-bound early stopping of experiments with sequential calibration. Chemical Engineering Research and Design, 230:756–766, 2026.

2. The theoretical guarantees are not convincingly connected to the stopping decisions that provide practical value.

Table 1 explicitly states that none of the replay configurations directly instantiates the corresponding certification theorem. Theorem 1 provides all-decision control, but its block-exclusive implementation does not stop in the reported experiments. Theorems 2 and 5 mainly concern the campaign's first stop, with qualifications involving the coupling between the deployed and reference systems. Yet Appendix C reports that 70%, 93%, and 76% of the weighted rule's channel-time savings on MATR, ISU-ILCC, and TONGJI, respectively, occur after the first stop.

The historical-pool weighted rule does stop some bad cells, but it remains audit-only. Moreover, the confirmatory configuration using an independently determined lifetime threshold abstains entirely on ISU-ILCC. These results leave a substantial gap between the settings where the theory applies and those where early stopping is useful.

I do not regard negative experimental results as a weakness in themselves. If the paper established that useful stopping with strong validity guarantees is inherently expensive, that would be an important finding. The current evidence, however, mainly establishes the limitations of the specific certification schemes examined. It does not sufficiently show that comparable costs are unavoidable for other valid stopping procedures. This weakens the paper's central conclusion about the practical price of statistical validity.

Questions for authors

Q1 — What is the main new theoretical insight beyond existing work on stratified and weighted conformal prediction?

Proposition 6 is one of the paper's clearest new results. Could the authors explain more precisely what it adds to Wieczorek's (2023) discussion of conformal prediction under stratified sampling? In particular, how does the failure of superuniformity established by the counterexample affect decisions at the paper's target level of α=0.1\alpha = 0.1, and what conclusions extend beyond the specific pooled statistic considered?

This distinction would materially affect my assessment of the paper's theoretical novelty and significance.

Reference

Jerzy Wieczorek. Design-based conformal prediction. Survey Methodology, 49(2):443–473, 2023.

Q2 — What does the evidence establish about the cost of obtaining useful validity guarantees?

The all-decision-certified construction never stops in the reported experiments, most of the practical weighted rule's savings occur after the first stop, and that weighted rule remains audit-only. Which theoretical or empirical result most clearly supports the conclusion that these costs reflect an inherent difficulty of safe closed-loop stopping, rather than mainly the particular certification schemes chosen here?

I am not asking for additional experiments as a general requirement. A clearer account of what the existing results do and do not establish would directly affect my assessment of the contribution.

7 comments

Dear Reviewer 9b42,

We sincerely appreciate your helpful comments and have addressed your concerns accordingly. As the discussion phase is about to close, we would be grateful for any further feedback you may wish to share.

Kind regards,
 The Authors

Thank you for the detailed rebuttal and substantial revisions. I appreciate the new testbed in Appendix D, which demonstrates that the certified methods can achieve meaningful stopping power when their assumptions hold, as well as the additional forecaster comparisons and clearer discussion of the scope of the theoretical results. These additions address important concerns from my original review.

However, my main concern about the scientific significance of the contribution remains unresolved.

The central question is how to achieve statistically valid, repeated early stopping under adaptive acquisition, and what this validity costs. The new experiments clarify the trade-offs of the proposed methods, but a substantial gap remains between their theoretical guarantees and their practical benefits. In Table 14, the all-decision-certified method achieves 0.79 recall on ISU-ILCC, but requires calibration channel time 4.18 times the campaign's own, while stopping recovers only 3% of that calibration cost. In Tables 13 and 16, the methods achieve meaningful overall recall, but their guarantees cover only the first stop, which accounts for a small fraction of the total time saved. The recommended weighted rule remains uncertified for repeated stopping under the evaluated stratified-pool setting. Table 15 provides a useful positive result for all-stop certification, but only after giving up adaptive acquisition, and its net savings disappear under end-of-life accounting.

My concern is not that these results are negative, but what general scientific conclusion they support. Proposition 6 provides a useful counterexample for the pooled conformal statistic under particular calibration compositions, but does not establish a limitation on alternative valid stopping procedures. The broader bounds in Propositions 5 and 7 largely instantiate established statistical results, while the substantial calibration costs, including Theorem 4's quadratic cost, are specific to the certification constructions considered. The revised paper now acknowledges these distinctions, which I appreciate.

I do not expect the paper to prove a universal lower bound. However, given the reliance on existing statistical ingredients, I would expect the new analysis to establish either a more broadly transferable insight into the difficulty of repeated safe stopping or a more compelling validity–resource trade-off in the adaptive setting. The current results mainly characterize the limitations and costs of the proposed designs. While technically meaningful, I do not find this sufficient to resolve my original concern about the overall significance and generality of the contribution.

The rebuttal substantially improves the paper, but does not change my overall assessment. I therefore maintain my rating of 2 (Weak rejection), with confidence 2.

AuthorsOpenReview ↗

Response to Reviewer 9b42's follow-up

Response to Reviewer 9b42's follow-up (1/3)

Thank you for the follow-up. For "a more broadly transferable insight into the difficulty of repeated safe stopping" or "a more compelling validity–resource trade-off", the revision adds an analysis of what each stop passes to the optimizer when one calibration pool is reused under adaptive acquisition (Appendix E, line 2538), and the conclusion now states its main result (line 539).

This analysis shows that, with a shared calibration pool, the difficulty of repeated safe stopping under adaptive acquisition comes from the stops themselves, which carry the pool's information to an optimizer that reads them and can break the guarantee after the first stop, while an optimizer reading only early forecasts keeps it at every stop (Proposition 8, line 2577; Corollary 1, line 2619). Neither proof uses any property of batteries (line 2549), and prior work does not cover this route through the stops, since there the calibration data are reused by an analyst, selected in a batch or kept independent by data splitting (remark (f), line 2746).

In the testbed experiment of Table 19, keeping the optimizer blind to stops certified every stop with the first-stop calibration pool alone, at recall close to that of an optimizer reading stops (line 2775), so the price of validity moves from calibration to optimizer regret.

The abstract (line 28) and the scope paragraph of Appendix E (line 2653) restrict the earlier cost statements to our adaptive acquisition, which reads stops.

Reviewer_QSADOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

Battery cycle life testing with certainty requires a lot of time during which physical channels are occupied. To circumvate this, testing frameworks were developed where a test is stopped early and a prediction is made on whether the battery is good or not, except that stopping the test destroys any evidence that the battery is good. While these methods save considerable time, they were found to report a significant amount of false negatives. The paper addresses this by introducing a testing framework that controls the error, with theoretical guarantees and experimental evidence. It consists of:

  1. predicting L~:=min⁡(L,Cx)\tilde{L}:=\min(L, C_x) where LL is the cell's life and CxC_x is the cell's recorded test horizon;
  2. decides using a statistical test whether the battery test should stop and decide the cell is bad;
  3. when a slot is free, randomly decide whether it should load a calibration cell or a train cell in order to restore exchangeability lost by adaptive acquisition. The paper provides a theoretical analysis of its framework that provides a guarantee of the confidence level under detailed technical assumptions. The authors stated the scope of these theoretical guarantees. Experiments demonstrate the precision of this testing framework.
Critical strengths
  1. The framework proposed by the paper could have a strong industrial impact by reducing testing time while providing accurate predictions compared to existing methods.

  2. The paper addresses itself some limitations of its framework and theoretical guarantees.

  3. Code is available with clear instructions for reproducibility.

Critical weaknesses
  1. The paper is not reader friendly. It has condensed expressions overloaded with technical terms, requiring careful reading multiple times. This holds for some statements of mathematical lemmas and theorems as well, some of which contain comments and remarks in the statement itself!

  2. Some statements or proofs in the appendix are not expressed mathematically, which can create ambiguity on mathematical statements due to reader interpretation. For example, in the statement of Lemma 2, it is not clear that "value-independent randomness" refers to the full value vector until one reads it in the proof.

Questions for authors

Can the authors rewrite the statements of the theorems and propositions in section 4 in order to improve readability, such that discussions, counterexamples and remarks are written separately?

2 comments

Dear Reviewer QSAD,

We sincerely appreciate your helpful comments and have addressed your concerns accordingly. As the discussion phase is about to close, we would be grateful for any further feedback you may wish to share.

Kind regards,
 The Authors

Dear Reviewer QSAD,

Thank you again for your review and for the time you have given to our submission. As the discussion period closes today, we would be grateful to know whether our replies and the revised PDF address your concerns. The revised PDF also adds Appendix E, which shows that every stop can be certified under an adaptive acquisition that does not read stops (Corollary 1 and Table 19). If our replies have resolved your concerns, we would be grateful if you would consider revisiting your score, and we would be glad to clarify anything that remains.

Kind regards, the Authors

Reviewer_qctYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at early stopping in closed-loop battery testing, where Bayesian optimization picks which designs to cycle and bad-looking tests get stopped early. Stopping a battery test can't be undone, so stopping a good cell loses the evidence that its design works. The authors want to bound the false-stop rate (the chance of stopping a cell that would have met the lifetime spec) with a finite-sample guarantee. The difficulty is that conformal prediction needs exchangeability, and the adaptive BO loop breaks it.

Their main idea is the ''ε\varepsilon-stream.'' Whenever a channel frees up, with probability ε\varepsilon it loads a calibration cell from a pre-registered random sequence. These calibration cells are kept away from the surrogate, forecaster and acquisition, so the weights for weighted conformal prediction are known exactly. The stopping rule uses a good-only conformal p-value with an α\alpha-spending schedule over a few looks, and it abstains when there's not enough calibration data.

The theory gives a ''ladder'' of guarantees, each with a stated cost:

  • exact control for all decisions with private calibration blocks;
  • control of only the first stop on a shared pool;
  • a front-loaded variant;
  • two classical lower bounds;
  • an impossibility result showing the weights used in practice can't get an exact distribution-free guarantee on stratified pools.

Experiments replay five public battery datasets. Successive halving falsely stops 47--96% of good cells. The weighted rule stays within α\alpha on every seed but has low recall (0.05 on MATR, 0.22 on ISU-ILCC). The certified modes almost never stop. The authors conclude that at realistic calibration sizes, a guarantee costs more channel time than early stopping saves.

Critical strengths
  • The problem is important and well motivated. The paper explains clearly why stopping a battery test is different from stopping a neural network training run, and the example of two cells with nearly the same capacity at cycle 100 but very different lifetimes is convincing.
  • The ε\varepsilon-stream is a simple idea. Making the conformal weights known by design, rather than estimating them, is a nice way around the exchangeability problem.
  • The paper is honest about its limits. It says clearly what each guarantee covers and what it costs, reports negative results, and admits the recommended rule is audit-only.
  • The experiments are thorough: multiple datasets, 100 seeds, worst-seed results in addition to means, many ablations and stress tests, and a fairly tuned LTT baseline.
  • Reproducibility is very strong: data hashes, bit-reproducible runs, and tables generated from result files.
Critical weaknesses
  • The guarantees don't cover the rule that's actually recommended. The authors say no replay row instantiates a theorem, and the recommended weighted rule is audit-only (Prop. 6 even shows it can't get an exact distribution-free guarantee on stratified pools). The modes that are covered are not useful: Thm. 1 never stops, Thm. 2 has recall 0.00--0.01, and Thm. 5 voids its certificate in 36% of campaigns. A simulation where all assumptions hold would help check the theorems and show how much calibration a certified mode needs to become useful.
  • The guarantees have a narrow scope. Thms. 2 and 5 only cover the first stop, but the paper says 70--93% of the saved channel time comes after it. The coupling factor in Thm. 2 is not measured, so it's unclear how close the bound is to α\alpha. Also, the control is per trial, not per campaign: 35/100 ISU-ILCC campaigns falsely kill at least one good cell.
  • The novelty is mostly in combining existing pieces. The statistic is from Jin & Cand`es (2026), good-only calibration from Bates et al.\ (2023), and reserving a random share of allocations from Liang & Bojinov (2024). The new parts (the closed-loop protocol, the coupling proofs, Prop. 6) should be separated more clearly in the main text, along with why Prinster et al.\ (2024) doesn't already cover this setting.
  • Only one forecaster is tried, but the main conclusion depends on it. Power is very forecaster-sensitive: on MATR, removing the ΔQ(V)\Delta Q(V) feature drops static recall from 0.529 to 0.045. Since the guarantees hold for any forecaster, trying stronger early-life predictors (e.g., Severson et al., 2019, or BatteryML models) would show whether the conclusion that certificates aren't worth it still holds.
Questions for authors

Please address the weaknesses above.

5 comments

Dear Reviewer qctY,

We sincerely appreciate your helpful comments and have addressed your concerns accordingly. As the discussion phase is about to close, we would be grateful for any further feedback you may wish to share.

Kind regards,
 The Authors

Dear Reviewer qctY,

Thank you again for your review and for the time you have given to our submission. As the discussion period closes today, we would be grateful to know whether our replies and the revised PDF address your concerns. The revised PDF also adds Appendix E, which shows that every stop can be certified under an adaptive acquisition that does not read stops (Corollary 1 and Table 19). If our replies have resolved your concerns, we would be grateful if you would consider revisiting your score, and we would be glad to clarify anything that remains.

Kind regards, the Authors

Dear Reviewer qctY,

Thank you again for your review. As the discussion period closes today, we would be glad to answer any remaining question about our replies or the revised PDF.

Kind regards, the Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.