- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Stopping a battery test early can save considerable time, but an incorrect decision also prevents the experiment from establishing whether a candidate design would have met its lifetime target. The central problem addressed here is how to control such false stops when subsequent experiments are selected adaptively using earlier results.
The proposed framework combines a frozen lifetime predictor, conformal calibration using known-good cells, and a separate -stream that collects calibration data independently of the acquisition process. The aim is to obtain known sampling probabilities and establish finite-sample guarantees for the stopping rule. The theoretical results cover different settings, including per-loaded-slot false-stop control across all looks under block-exclusive calibration and first-stop control under shared or front-loaded calibration, subject to the stated coupling qualifications, together with analyses of calibration granularity, sample reuse, and resource costs.
Experiments on five public battery datasets, together with additional case studies, show that aggressive stopping can eliminate many good cells. However, the block-exclusive configuration associated with the strongest all-decision guarantee does not stop in the reported replays, while the more practical weighted historical-pool rule remains audit-only. The authors conclude that the cost of stronger certification can exceed the testing time saved under the evaluated conditions.
Critical strengths
1. The paper addresses a meaningful consequence of early stopping that standard efficiency measures can overlook.
A prematurely stopped battery test cannot simply be resumed, and stopping a good cell can mean losing the evidence needed to identify a successful design. The paper makes this distinction concrete by measuring false stops and successful observed endpoints, rather than treating every reduction in testing time as a benefit. The replay experiments also show why this matters: aggressive halving-based stopping can discard a large fraction of good cells. This is a useful problem formulation for expensive physical experiments.
2. The paper provides concrete examples of where statistical guarantees do and do not apply.
Proposition 6 gives an explicit counterexample showing that the proposed pooled conformal statistic need not be valid under fixed stratified calibration, even when the individual importance weights are known. The analysis also distinguishes guarantees for fixed physical cells from those for adaptively selected loading slots, and makes clear that first-stop control is not the same as controlling every subsequent stop. These are useful technical distinctions. Although I question the overall significance of the new results, they provide a substantive basis for the work.
Critical weaknesses
1. The theoretical results do not yet establish a sufficiently important advance over the closest prior work.
The paper raises a reasonable question, but several of the statistical difficulties at its center have already been studied. Wieczorek (2023) explains why standard covariate-shift conformal arguments do not directly carry over to stratified sampling. Tibshirani et al. (2019) establish weighted conformal prediction under distribution shift, while Fannjiang et al. (2022) extend the analysis to feedback-driven data selection. Calibrated false-stop control for expensive experiments has also been investigated by Xiao et al. (2026), although that work does not address the same closed-loop setting.
There are new results here, particularly Proposition 6, which gives an explicit counterexample for the pooled statistic used in the paper. However, its conclusion concerns that statistic under particular sampling designs; it does not rule out other valid approaches. Likewise, some of the reported limitations follow from established statistical bounds, while others depend on the proposed certification schemes. For example, Theorem 4's quadratic calibration cost applies to a particular class of private-block procedures certified through a union bound, rather than to repeated safe stopping in general.
The authors do distinguish these different kinds of limitations, which I appreciate. My concern remains that, once those qualifications and the closest prior work are taken into account, the central new insight is less substantial than the paper's emphasis on the "price of validity" suggests. I am not convinced that the analysis establishes a sufficiently important new limit or design principle for safe closed-loop stopping.
References
Clara Fannjiang, Stephen Bates, Anastasios N. Angelopoulos, Jennifer Listgarten, and Michael I. Jordan. Conformal prediction under feedback covariate shift for biomolecular design. Proceedings of the National Academy of Sciences, 119(43), 2022.
Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems, volume 32, pp. 2530–2540, 2019.
Jerzy Wieczorek. Design-based conformal prediction. Survey Methodology, 49(2):443–473, 2023.
Yongrui Xiao, Mengjia Zhu, Yu Zhang, Dimitrios Tsaoulidis, and Tao Chen. Confidence-bound early stopping of experiments with sequential calibration. Chemical Engineering Research and Design, 230:756–766, 2026.
2. The theoretical guarantees are not convincingly connected to the stopping decisions that provide practical value.
Table 1 explicitly states that none of the replay configurations directly instantiates the corresponding certification theorem. Theorem 1 provides all-decision control, but its block-exclusive implementation does not stop in the reported experiments. Theorems 2 and 5 mainly concern the campaign's first stop, with qualifications involving the coupling between the deployed and reference systems. Yet Appendix C reports that 70%, 93%, and 76% of the weighted rule's channel-time savings on MATR, ISU-ILCC, and TONGJI, respectively, occur after the first stop.
The historical-pool weighted rule does stop some bad cells, but it remains audit-only. Moreover, the confirmatory configuration using an independently determined lifetime threshold abstains entirely on ISU-ILCC. These results leave a substantial gap between the settings where the theory applies and those where early stopping is useful.
I do not regard negative experimental results as a weakness in themselves. If the paper established that useful stopping with strong validity guarantees is inherently expensive, that would be an important finding. The current evidence, however, mainly establishes the limitations of the specific certification schemes examined. It does not sufficiently show that comparable costs are unavoidable for other valid stopping procedures. This weakens the paper's central conclusion about the practical price of statistical validity.
Questions for authors
Q1 — What is the main new theoretical insight beyond existing work on stratified and weighted conformal prediction?
Proposition 6 is one of the paper's clearest new results. Could the authors explain more precisely what it adds to Wieczorek's (2023) discussion of conformal prediction under stratified sampling? In particular, how does the failure of superuniformity established by the counterexample affect decisions at the paper's target level of , and what conclusions extend beyond the specific pooled statistic considered?
This distinction would materially affect my assessment of the paper's theoretical novelty and significance.
Reference
Jerzy Wieczorek. Design-based conformal prediction. Survey Methodology, 49(2):443–473, 2023.
Q2 — What does the evidence establish about the cost of obtaining useful validity guarantees?
The all-decision-certified construction never stops in the reported experiments, most of the practical weighted rule's savings occur after the first stop, and that weighted rule remains audit-only. Which theoretical or empirical result most clearly supports the conclusion that these costs reflect an inherent difficulty of safe closed-loop stopping, rather than mainly the particular certification schemes chosen here?
I am not asking for additional experiments as a general requirement. A clearer account of what the existing results do and do not establish would directly affect my assessment of the contribution.
7 comments
Thank you for the detailed rebuttal and substantial revisions. I appreciate the new testbed in Appendix D, which demonstrates that the certified methods can achieve meaningful stopping power when their assumptions hold, as well as the additional forecaster comparisons and clearer discussion of the scope of the theoretical results. These additions address important concerns from my original review.
However, my main concern about the scientific significance of the contribution remains unresolved.
The central question is how to achieve statistically valid, repeated early stopping under adaptive acquisition, and what this validity costs. The new experiments clarify the trade-offs of the proposed methods, but a substantial gap remains between their theoretical guarantees and their practical benefits. In Table 14, the all-decision-certified method achieves 0.79 recall on ISU-ILCC, but requires calibration channel time 4.18 times the campaign's own, while stopping recovers only 3% of that calibration cost. In Tables 13 and 16, the methods achieve meaningful overall recall, but their guarantees cover only the first stop, which accounts for a small fraction of the total time saved. The recommended weighted rule remains uncertified for repeated stopping under the evaluated stratified-pool setting. Table 15 provides a useful positive result for all-stop certification, but only after giving up adaptive acquisition, and its net savings disappear under end-of-life accounting.
My concern is not that these results are negative, but what general scientific conclusion they support. Proposition 6 provides a useful counterexample for the pooled conformal statistic under particular calibration compositions, but does not establish a limitation on alternative valid stopping procedures. The broader bounds in Propositions 5 and 7 largely instantiate established statistical results, while the substantial calibration costs, including Theorem 4's quadratic cost, are specific to the certification constructions considered. The revised paper now acknowledges these distinctions, which I appreciate.
I do not expect the paper to prove a universal lower bound. However, given the reliance on existing statistical ingredients, I would expect the new analysis to establish either a more broadly transferable insight into the difficulty of repeated safe stopping or a more compelling validity–resource trade-off in the adaptive setting. The current results mainly characterize the limitations and costs of the proposed designs. While technically meaningful, I do not find this sufficient to resolve my original concern about the overall significance and generality of the contribution.
The rebuttal substantially improves the paper, but does not change my overall assessment. I therefore maintain my rating of 2 (Weak rejection), with confidence 2.
Response to Reviewer 9b42's follow-up
Response to Reviewer 9b42's follow-up (1/3)
Thank you for the follow-up. For "a more broadly transferable insight into the difficulty of repeated safe stopping" or "a more compelling validity–resource trade-off", the revision adds an analysis of what each stop passes to the optimizer when one calibration pool is reused under adaptive acquisition (Appendix E, line 2538), and the conclusion now states its main result (line 539).
This analysis shows that, with a shared calibration pool, the difficulty of repeated safe stopping under adaptive acquisition comes from the stops themselves, which carry the pool's information to an optimizer that reads them and can break the guarantee after the first stop, while an optimizer reading only early forecasts keeps it at every stop (Proposition 8, line 2577; Corollary 1, line 2619). Neither proof uses any property of batteries (line 2549), and prior work does not cover this route through the stops, since there the calibration data are reused by an analyst, selected in a batch or kept independent by data splitting (remark (f), line 2746).
In the testbed experiment of Table 19, keeping the optimizer blind to stops certified every stop with the first-stop calibration pool alone, at recall close to that of an optimizer reading stops (line 2775), so the price of validity moves from calibration to optimizer regret.
The abstract (line 28) and the scope paragraph of Appendix E (line 2653) restrict the earlier cost statements to our adaptive acquisition, which reads stops.
Dear Reviewer 9b42,
We sincerely appreciate your helpful comments and have addressed your concerns accordingly. As the discussion phase is about to close, we would be grateful for any further feedback you may wish to share.
Kind regards, The Authors