- Rating
- 3
- Confidence
- 2(−1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies an important and timely problem: how to audit reliably with minimal resources. Specifically, the method asks both the candidate and a trusted reference model to output one symbol from a fixed alphabet. Repeated responses yield empirical categorical distributions, which we compare using total variation (TV) distance and a paired permutation test.
Critical strengths
- The paper studies an important and interesting problem.
- The experimental results demonstrates that the method achieves strong performance at substantially lower cost than existing SOTA methods.
Critical weaknesses
-
The Methodology in Section 4 introduces numerous definitions and theorem, making it difficult to understand and follow.
-
My main concern is how to construct the input query. Section 4.2 shows that when Δ > 0, there exists a query that induces different output distributions for two models. However, how can such a query be identified in practice? Section directly uses a fixed query instruction: "Choose a random lowercase English letter from a to z. Output exactly that one letter and nothing else". What motivates this particular choice, and how does it relate to the theoretical analysis in Section 4.2?
Questions for authors
- My main concerns is the connection between the theoretical analysis in Section 4.2, and the practical implementation in Section 4.1. Section 4.2 establishes the existence of a distinguishing query but does not explain how to identify such a query in practice. In fact, the proposed method directly adopts a fixed query instruction. I think it is logically necessary to clarify whether the theory provides any guidance for query design.
Response to Reviewer WF89
Thank you so much for your encouraging review! We were really happy to see your recognition of the problem’s importance and the empirical results. We also appreciate your thoughtful comments on the theory and presentation. These comments helped us identify where the connection to our practical method needed to be clearer.
W1. Presentation of the methodology and theoretical analysis
Thank you for pointing this out. We have clarified the organization of Section 4.2 by separating the analysis into two explicit questions:
This distinction separates the structural existence result from the guarantee for the practical testing procedure in Section 4.1. The new result is stated as Theorem 4.5, with its full proof in Appendix B. We hope this makes the purpose of each theoretical result easier to follow.
W2. Motivation for the practical query and how it is chosen
The lowercase-letter instruction is a cost-motivated design choice. It allows a one-token generation cap and produces a small parsed category space consisting of letters and INVALID. The same instruction can also be reused across model pairs without a separate search for model-specific queries.
Our choice is supported empirically by the prompt ablation in Figure 5, which compares the instruction alone with an UltraChat message followed by the same instruction. Under the evaluated settings, the instruction alone provides the better observed cost–power trade-off. We do not assume that the models actually sample letters uniformly; the test compares their induced response distributions.
Accordingly, Theorem 4.4 and our practical prompt now play clear, complementary roles: Theorem 4.4 establishes the theoretical principle that single-token responses can carry strong distinguishing signal, while our empirical ablations motivate the fixed one-character instruction as a simple, zero-search, cost-efficient instantiation.
Q1. Connecting the theoretical analysis to the practical method
To address this connection directly, we have added Theorem 4.5 in Section 4.2.2, with the full proof in Appendix B.
The new theorem analyzes the paired TV test for a fixed prompt, parser, and endpoint configuration, including its finite Monte Carlo implementation. Under independent sampling, it provides:
These guarantees do not require the discrete-attention architecture used in Theorem 4.4. They apply to the fixed response distributions being tested, which directly connects the analysis to our practical procedure.
The power guarantee is conditional on a positive response-distribution gap. It does not assert that the chosen prompt distinguishes every model pair: if two models induce the same parsed distribution at that prompt, repeating it cannot distinguish them. This clarifies both what the practical test can establish and the limitation of using a fixed query.
We sincerely appreciate your feedback, which motivated this additional theoretical analysis. With the new theorem, its proof, and the clearer distinction between the two theoretical questions now included in the revised PDF, we hope you will consider raising your score if these revisions address your concerns. Thank you again for your time and encouragement!