- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper is focused on "when to stop testing an AI system" by estimating the probability of upper bound of meeting an unseen failure. It proposes a mixture of restarted betting processes and extends the framework to known distribution shift and imperfect failure labeling. Experiments use synthetic data and repeated sampling from an annotated SWE-Chat dataset.
Critical strengths
- The topic is interesting, and the setting is close to the abstract workflow of how people testing systems.
- The theory part seems to be sound.
Critical weaknesses
- The paper uses much narrower margins than the official ICLR template. This allows more content within the page limit.
- I think the setting is over-simplified. The most critical part in real-world testing is how to ensure the test-coverage. The whole paper is based on an assumption that the more you test, the more coverage you have. However, in the real-world, this would rarely happen. A simple example is that you tried multiple tests with integer input for an system, but the bug only occurs when inputs are floats. This problem is classic in software testing scope (see "STADS: Software Testing as Species Discovery" TOSEM, 2018), which also discussed the "when-to-stop-testing" topic. In agent system testing scenario, since the input space can be very large, I don't think such complexity can be easily ignored.
- The paper's scope is unclear to me. It looks like a pure theory paper where only formulations in the methods without any real-world application example of "agent system". But the title claimed a relationship to the "agent system" scenario to me. If it really related to "agent system testing", I think at least STADS should be discussed or compared in the paper.
- From my understanding, the quality of the experiment highly depends on the definition of "unseen fail". The labels seems all come from LLM judge. I think it matters since if we classify all failures into the same type, them the bound drops to zero right after finding the first failure.
Questions for authors
- Is this a pure theory paper?
- Can you use the correct paper format?
- Can you compare or at least discuss the difference between this paper and STADS?
On the margins of the paper. Thank you for catching the margins issue. (A spurious package import had changed the margins.) Our updated manuscript has the correct margins. We moved some of the background on SAVI, discussion of intuition for warm-starts and choices of the hyper parameter lambda, and discussion of the technical challenges of our results, to the appendix. We also moved some of the empirical setup details to the appendix.
On the connection to software testing. We disagree with the claim that our “assumption that the more you test, the more coverage you have”. Instead, the analogous statement in our setting is “the more you observe real-world usage of a system, the more coverage with respect to that distribution you have”. We’ve added a discussion of STADs and relation to software testing to the paper; their results (and those in the software testing literature) are not anytime valid and can therefore not be used for optimal stopping while preserving statistical validity. Our setting is different from classic software testing because with AI systems the goal is not test-coverage (this may be impossible with output spaces as large as AI systems!). Instead, the goal is a guarantee with respect to a distribution of usage for a population of beta testers or historical usage data.
On the real-world application. The paper’s primary contributions are the method and associated theory guarantees. The empirics are in support of establishing the finite-sample data efficiency of our method compared to other sensible approaches. We instantiate the method on real AI-agent outputs, but the goal of the empirics is to validate the method in circumstances in which a “ground-truth” missing mass can be observed. In real applications, no such ground-truth missing mass is available. Thus, we try to use a setting that is as close to a real operational setting as possible without losing our ability to measure ground truth and compare to other approaches.
On the importance of the unseen and fail functions. We agree (and discuss in the paper, see the first sentence of limitations in the discussion section) that these definitions are very important in practice and depend on the deployment context. Our contribution solves what we believe to be an important and interesting technical question of how to stop testing once the definitions of unseen and fail have been fixed and determined to be relevant to the deployment decision at hand. Future work examining how to choose these definitions would be valuable but is inherently context-dependent and should not be expected to be resolved in our work (and definitely not at the level of abstraction at which we are able to solve the optimal/optional stopping question), which proposes and establishes guarantees once these definitions have been satisfactorily determined. Our method (just like any statistical estimator) will not yield useful results if the quantity being measured (e.g., a trivial fail function that never marks anything as a failure) is very different from the real-world phenomenon it is intended to capture.