- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper aims to address two important auditing problems theoretically and empirically: 'First, a difference found during search may not hold on a broader population of inputs, especially because search favors large observed differences. Second, a search that finds no difference may still miss differences on inputs that were not tested.' The important experiments E1, E2 and E3 are used to support their theory/framework, and they cover 'search-selected differences through fresh validation', 'the validation rule under controlled population truth', and 'negative search outcomes'.
Critical strengths
This paper focuses on the model replacement auditing problem, which is a relevant research challenge. The authors address this problem theoretically and also test their framework on a function-calling benchmark BFCL and in controlled simulations. The theoretical proofs (Theorem 1, Props 1 and 2) appear sound. I used Claude Fable 5.1 to help me check them and found no errors.
Critical weaknesses
-
The research challenge is not well presented. It lacks a clear problem statement section in Sec. 2, and many invented terms in this specific research case, such as 'contracts' and 'kernels', are only briefly defined, and after their first use. Also, no related work section is given to help the readers dive into this topic. This is consequential because, without that positioning, it is hard to see what is new: the validation procedure is a standard holdout with a Hoeffding bound.
-
The practical usage of the proposed theory/framework is not well addressed. The three experiments E1-3 are proposed to support the two studied research questions, but do not highlight their real usage in applications. Replacement in E1 is a convex combination of peer-model scores rather than an actual candidate model, and the negative-search result requires a Lipschitz constant that the paper admits does not hold on its real data.
Given the two points above, it is hard to estimate how much impact or effectiveness this framework could have for future research in auditing.
Questions for authors
-
Is there any related work that connects to the invented vocabulary in this work (e.g., contract, regime, kernel)?
-
What do the legends in Figure 1 mean? The axes and markers are only defined in Section 3.
-
E1-3 are mentioned in the introduction without giving any details, so it does not help to understand the paper. Together with question 2, the main motivation needs to be presented in a better way.
-
Why is the replacement in E1 a convex combination of peer-model scores rather than an actual candidate model?
-
The negative-search result needs a Lipschitz constant L, which the paper admits does not hold on the real data. How can L be estimated in practice?